[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:03:46 dist-test-slave-1l3l krb5kdc[5459](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:03:46 dist-test-slave-1l3l krb5kdc[5459](info): set up 2 sockets
Aug 08 04:03:46 dist-test-slave-1l3l krb5kdc[5459](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260808 04:03:47.894901 5452 mini_kdc.cc:121] Time spent starting KDC: real 1.176s user 0.006s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/krb5kdc/test-user.keytab.
Aug 08 04:03:48 dist-test-slave-1l3l krb5kdc[5459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:03:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:03:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.83.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab.
Entry for principal kudu/127.5.83.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab.
I20260808 04:03:48.064256 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:41109
--webserver_interface=127.5.83.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab
--principal=kudu/127.5.83.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:37819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.62:41109
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:03:48.418872 5479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:03:48.419265 5479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:03:48.419412 5479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:03:48.426452 5479 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:03:48.426563 5479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:03:48.426645 5479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:03:48.426730 5479 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:03:48.426789 5479 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:03:48.436724 5479 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:37819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.62:41109
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab
--principal=kudu/127.5.83.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:41109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:03:48.438655 5479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:03:48.440665 5479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:03:48.461694 5492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:03:48.461737 5488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:03:48.461652 5487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:03:48.465132 5479 server_base.cc:1034] running on GCE node
I20260808 04:03:48.466089 5479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:03:48.468149 5479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:03:48.469357 5479 hybrid_clock.cc:648] HybridClock initialized: now 1786161828469254 us; error 89 us; skew 500 ppm
I20260808 04:03:48.469835 5479 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:03:48 dist-test-slave-1l3l krb5kdc[5459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161828, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:03:48.473049 5479 init.cc:377] Logged in from keytab as kudu/127.5.83.62@KRBTEST.COM (short username kudu)
I20260808 04:03:48.475397 5479 webserver.cc:460] Webserver started at http://127.5.83.62:38507/ using document root <none> and password file <none>
I20260808 04:03:48.476320 5479 fs_manager.cc:359] Metadata directory not provided
I20260808 04:03:48.476446 5479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:03:48.476732 5479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:03:48.481351 5479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/minicluster-data/master-0/data/instance:
uuid: "1bc3b32f6efb46a6afe75bc5d474acbd"
format_stamp: "Formatted at 2026-08-08 04:03:48 on dist-test-slave-1l3l"
server_key: "f1c1effaf7addb710c2665ab003751d8"
server_key_iv: "960d5905986adb18cb4004fb8b9e9656"
server_key_version: "encryptionkey@0"
I20260808 04:03:48.482110 5479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/minicluster-data/master-0/wal/instance:
uuid: "1bc3b32f6efb46a6afe75bc5d474acbd"
format_stamp: "Formatted at 2026-08-08 04:03:48 on dist-test-slave-1l3l"
server_key: "f1c1effaf7addb710c2665ab003751d8"
server_key_iv: "960d5905986adb18cb4004fb8b9e9656"
server_key_version: "encryptionkey@0"
I20260808 04:03:48.488265 5479 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260808 04:03:48.492059 5501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:03:48.493420 5479 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:03:48.493662 5479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/minicluster-data/master-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/minicluster-data/master-0/wal
uuid: "1bc3b32f6efb46a6afe75bc5d474acbd"
format_stamp: "Formatted at 2026-08-08 04:03:48 on dist-test-slave-1l3l"
server_key: "f1c1effaf7addb710c2665ab003751d8"
server_key_iv: "960d5905986adb18cb4004fb8b9e9656"
server_key_version: "encryptionkey@0"
I20260808 04:03:48.493839 5479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:03:48.516515 5479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:03:48.521401 5479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:03:48.521713 5479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:03:48.542433 5479 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.62:41109
I20260808 04:03:48.542464 5575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.62:41109 every 8 connection(s)
I20260808 04:03:48.544751 5479 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
I20260808 04:03:48.550246 5576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:03:48.550577 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 5479
I20260808 04:03:48.550988 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/minicluster-data/master-0/wal/instance
I20260808 04:03:48.553644 5452 external_mini_cluster.cc:1428] Setting key dbebc5d0dd87f15b260c4f812a1d7bf2
I20260808 04:03:48.565805 5576 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bc3b32f6efb46a6afe75bc5d474acbd: Bootstrap starting.
I20260808 04:03:48.569203 5576 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1bc3b32f6efb46a6afe75bc5d474acbd: Neither blocks nor log segments found. Creating new log.
I20260808 04:03:48.570621 5576 log.cc:824] T 00000000000000000000000000000000 P 1bc3b32f6efb46a6afe75bc5d474acbd: Log is configured to *not* fsync() on all Append() calls
I20260808 04:03:48.574323 5576 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bc3b32f6efb46a6afe75bc5d474acbd: No bootstrap required, opened a new log
I20260808 04:03:48.583082 5576 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1bc3b32f6efb46a6afe75bc5d474acbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bc3b32f6efb46a6afe75bc5d474acbd" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 41109 } }
I20260808 04:03:48.583582 5576 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1bc3b32f6efb46a6afe75bc5d474acbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:03:48.583779 5576 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1bc3b32f6efb46a6afe75bc5d474acbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bc3b32f6efb46a6afe75bc5d474acbd, State: Initialized, Role: FOLLOWER
I20260808 04:03:48.584852 5576 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1bc3b32f6efb46a6afe75bc5d474acbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bc3b32f6efb46a6afe75bc5d474acbd" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 41109 } }
I20260808 04:03:48.585160 5576 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1bc3b32f6efb46a6afe75bc5d474acbd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:03:48.585455 5576 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1bc3b32f6efb46a6afe75bc5d474acbd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:03:48.585712 5576 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1bc3b32f6efb46a6afe75bc5d474acbd [term 0 FOLLOWER]: Advancing to term 1
Aug 08 04:03:48 dist-test-slave-1l3l krb5kdc[5459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:03:48.588125 5576 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1bc3b32f6efb46a6afe75bc5d474acbd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bc3b32f6efb46a6afe75bc5d474acbd" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 41109 } }
I20260808 04:03:48.589176 5576 leader_election.cc:302] T 00000000000000000000000000000000 P 1bc3b32f6efb46a6afe75bc5d474acbd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1bc3b32f6efb46a6afe75bc5d474acbd; no voters:
I20260808 04:03:48.590044 5576 leader_election.cc:288] T 00000000000000000000000000000000 P 1bc3b32f6efb46a6afe75bc5d474acbd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:03:48.590317 5581 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1bc3b32f6efb46a6afe75bc5d474acbd [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:03:48.591462 5581 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1bc3b32f6efb46a6afe75bc5d474acbd [term 1 LEADER]: Becoming Leader. State: Replica: 1bc3b32f6efb46a6afe75bc5d474acbd, State: Running, Role: LEADER
I20260808 04:03:48.592374 5581 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1bc3b32f6efb46a6afe75bc5d474acbd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bc3b32f6efb46a6afe75bc5d474acbd" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 41109 } }
I20260808 04:03:48.592869 5576 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1bc3b32f6efb46a6afe75bc5d474acbd [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:03:48.599082 5583 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bc3b32f6efb46a6afe75bc5d474acbd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1bc3b32f6efb46a6afe75bc5d474acbd. Latest consensus state: current_term: 1 leader_uuid: "1bc3b32f6efb46a6afe75bc5d474acbd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bc3b32f6efb46a6afe75bc5d474acbd" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 41109 } } }
I20260808 04:03:48.599118 5582 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bc3b32f6efb46a6afe75bc5d474acbd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1bc3b32f6efb46a6afe75bc5d474acbd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bc3b32f6efb46a6afe75bc5d474acbd" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 41109 } } }
I20260808 04:03:48.599591 5583 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bc3b32f6efb46a6afe75bc5d474acbd [sys.catalog]: This master's current role is: LEADER
I20260808 04:03:48.599596 5582 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bc3b32f6efb46a6afe75bc5d474acbd [sys.catalog]: This master's current role is: LEADER
I20260808 04:03:48.600426 5578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:48.560426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48886 (local address 127.5.83.62:41109)
0808 04:03:48.561666 (+ 1240us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:48.561683 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:48.564213 (+ 2530us) server_negotiation.cc:407] Connection header received
0808 04:03:48.573536 (+ 9323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:48.573556 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:48.574076 (+ 520us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:48.574466 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:48.577589 (+ 3123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:48.578962 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:48.580679 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:48.581403 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:48.581665 (+ 262us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:48.587223 (+ 5558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:48.587287 (+ 64us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:48.587301 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:03:48.587344 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:48.591290 (+ 3946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:48.592463 (+ 1173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:48.592471 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:48.592479 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:48.592742 (+ 263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:48.593306 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:48.593311 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:48.593314 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:48.594135 (+ 821us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:48.597677 (+ 3542us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:48.597977 (+ 300us) server_negotiation.cc:299] Negotiation successful
0808 04:03:48.598388 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":838,"thread_start_us":417,"threads_started":1}
I20260808 04:03:48.607038 5594 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:03:48.613670 5594 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:03:48.625218 5594 catalog_manager.cc:1347] Generated new cluster ID: 0e513bc426144971922b8870f3798d2c
I20260808 04:03:48.625339 5594 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:03:48.643935 5594 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:03:48.645452 5594 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:03:48.654474 5594 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1bc3b32f6efb46a6afe75bc5d474acbd: Generated new TSK 0
I20260808 04:03:48.655500 5594 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.83.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.1.keytab.
Entry for principal kudu/127.5.83.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.1.keytab.
I20260808 04:03:48.709378 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.1:0
--local_ip_for_outbound_sockets=127.5.83.1
--webserver_interface=127.5.83.1
--webserver_port=0
--tserver_master_addrs=127.5.83.62:41109
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.1.keytab
--principal=kudu/127.5.83.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:37819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:03:49.025136 5608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:03:49.025410 5608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:03:49.025549 5608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:03:49.034907 5608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:03:49.035023 5608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:03:49.035184 5608 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.1
I20260808 04:03:49.044726 5608 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:37819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.1.keytab
--principal=kudu/127.5.83.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.83.1
--webserver_port=0
--tserver_master_addrs=127.5.83.62:41109
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.1
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:03:49.046041 5608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:03:49.047360 5608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:03:49.060782 5618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:03:49.060817 5621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:03:49.060868 5619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:03:49.061623 5608 server_base.cc:1034] running on GCE node
I20260808 04:03:49.062192 5608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:03:49.063055 5608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:03:49.064244 5608 hybrid_clock.cc:648] HybridClock initialized: now 1786161829064179 us; error 61 us; skew 500 ppm
I20260808 04:03:49.064533 5608 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:03:49 dist-test-slave-1l3l krb5kdc[5459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161829, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:03:49.067514 5608 init.cc:377] Logged in from keytab as kudu/127.5.83.1@KRBTEST.COM (short username kudu)
I20260808 04:03:49.068915 5608 webserver.cc:460] Webserver started at http://127.5.83.1:42243/ using document root <none> and password file <none>
I20260808 04:03:49.069536 5608 fs_manager.cc:359] Metadata directory not provided
I20260808 04:03:49.069653 5608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:03:49.069939 5608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:03:49.072752 5608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/minicluster-data/ts-0/data/instance:
uuid: "b345af3cc1434a5da98ecb2d37ae9589"
format_stamp: "Formatted at 2026-08-08 04:03:49 on dist-test-slave-1l3l"
server_key: "18d45b15ec1073497f1b6807b94a1dbe"
server_key_iv: "4f933571a4e7fcd7ce32ee8917a9dae1"
server_key_version: "encryptionkey@0"
I20260808 04:03:49.073632 5608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/minicluster-data/ts-0/wal/instance:
uuid: "b345af3cc1434a5da98ecb2d37ae9589"
format_stamp: "Formatted at 2026-08-08 04:03:49 on dist-test-slave-1l3l"
server_key: "18d45b15ec1073497f1b6807b94a1dbe"
server_key_iv: "4f933571a4e7fcd7ce32ee8917a9dae1"
server_key_version: "encryptionkey@0"
I20260808 04:03:49.078703 5608 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260808 04:03:49.082069 5631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:03:49.083305 5608 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260808 04:03:49.083469 5608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/minicluster-data/ts-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/minicluster-data/ts-0/wal
uuid: "b345af3cc1434a5da98ecb2d37ae9589"
format_stamp: "Formatted at 2026-08-08 04:03:49 on dist-test-slave-1l3l"
server_key: "18d45b15ec1073497f1b6807b94a1dbe"
server_key_iv: "4f933571a4e7fcd7ce32ee8917a9dae1"
server_key_version: "encryptionkey@0"
I20260808 04:03:49.083648 5608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:03:49.101944 5608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:03:49.105666 5608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:03:49.105913 5608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:03:49.106938 5608 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:03:49.108600 5608 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:03:49.108680 5608 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:03:49.108872 5608 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:03:49.108940 5608 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:03:49.140631 5608 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.1:42913
I20260808 04:03:49.140666 5792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.1:42913 every 8 connection(s)
I20260808 04:03:49.141850 5608 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
I20260808 04:03:49.145515 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 5608
I20260808 04:03:49.145721 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/minicluster-data/ts-0/wal/instance
I20260808 04:03:49.146318 5452 external_mini_cluster.cc:1428] Setting key 32fe713fc63a59635531422d93603794
Aug 08 04:03:49 dist-test-slave-1l3l krb5kdc[5459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161829, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.1@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:03:49.163200 5795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:49.145074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.1:40061 (local address 127.5.83.62:41109)
0808 04:03:49.145562 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:49.145568 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:49.146636 (+ 1068us) server_negotiation.cc:407] Connection header received
0808 04:03:49.148942 (+ 2306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:49.148948 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:49.149026 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:49.149146 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:49.150595 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:49.151580 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:49.153141 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:49.153715 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:49.153808 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:49.157736 (+ 3928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:49.157741 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:49.157744 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:03:49.157775 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:49.160243 (+ 2468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:49.160929 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:49.160935 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:49.160939 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:49.161063 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:49.161471 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:49.161475 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:49.161477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:49.161750 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:49.161881 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:49.162615 (+ 734us) server_negotiation.cc:299] Negotiation successful
0808 04:03:49.162831 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":234,"threads_started":1}
WARNING: no policy specified for kudu/127.5.83.2@KRBTEST.COM; defaulting to no policy
I20260808 04:03:49.164180 5796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:49.145596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:41109 (local address 127.5.83.1:40061)
0808 04:03:49.146460 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:49.146506 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 04:03:49.148675 (+ 2169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:03:49.149373 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:03:49.149381 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:03:49.149863 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:03:49.150424 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:49.150441 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:49.151773 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:49.151779 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:49.153006 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:49.153019 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:49.153952 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:49.153960 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:49.154280 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:49.154994 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:03:49.155030 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:03:49.157526 (+ 2496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:03:49.160371 (+ 2845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:03:49.160380 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:03:49.160396 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:03:49.160804 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:03:49.161168 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:03:49.161173 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:03:49.161177 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:03:49.161358 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:03:49.161872 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:03:49.161888 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:03:49.162195 (+ 307us) client_negotiation.cc:769] Sending connection context
0808 04:03:49.162506 (+ 311us) client_negotiation.cc:240] Negotiation successful
0808 04:03:49.163026 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":344,"threads_started":1}
I20260808 04:03:49.165917 5793 heartbeater.cc:347] Connected to a master server at 127.5.83.62:41109
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.2@KRBTEST.COM" created.
I20260808 04:03:49.166429 5793 heartbeater.cc:464] Registering TS with master...
I20260808 04:03:49.167271 5793 heartbeater.cc:511] Master 127.5.83.62:41109 requested a full tablet report, sending...
I20260808 04:03:49.171525 5522 ts_manager.cc:194] Registered new tserver with Master: b345af3cc1434a5da98ecb2d37ae9589 (127.5.83.1:42913)
I20260808 04:03:49.173688 5522 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.1@KRBTEST.COM'} at 127.5.83.1:40061
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.2.keytab.
Entry for principal kudu/127.5.83.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.2.keytab.
I20260808 04:03:49.188995 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.2:0
--local_ip_for_outbound_sockets=127.5.83.2
--webserver_interface=127.5.83.2
--webserver_port=0
--tserver_master_addrs=127.5.83.62:41109
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.2.keytab
--principal=kudu/127.5.83.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:37819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:03:49.503402 5801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:03:49.503659 5801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:03:49.503813 5801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:03:49.510667 5801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:03:49.510778 5801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:03:49.510936 5801 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.2
I20260808 04:03:49.520092 5801 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:37819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.2.keytab
--principal=kudu/127.5.83.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.83.2
--webserver_port=0
--tserver_master_addrs=127.5.83.62:41109
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.2
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:03:49.521232 5801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:03:49.522478 5801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:03:49.534564 5808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:03:49.534615 5809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:03:49.534641 5812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:03:49.535519 5801 server_base.cc:1034] running on GCE node
I20260808 04:03:49.536073 5801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:03:49.536911 5801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:03:49.538085 5801 hybrid_clock.cc:648] HybridClock initialized: now 1786161829538036 us; error 46 us; skew 500 ppm
I20260808 04:03:49.538350 5801 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:03:49 dist-test-slave-1l3l krb5kdc[5459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161829, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:03:49.541236 5801 init.cc:377] Logged in from keytab as kudu/127.5.83.2@KRBTEST.COM (short username kudu)
I20260808 04:03:49.542651 5801 webserver.cc:460] Webserver started at http://127.5.83.2:34095/ using document root <none> and password file <none>
I20260808 04:03:49.543299 5801 fs_manager.cc:359] Metadata directory not provided
I20260808 04:03:49.543411 5801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:03:49.543691 5801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:03:49.546389 5801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/minicluster-data/ts-1/data/instance:
uuid: "c48da412664045f986366a3eb75b116a"
format_stamp: "Formatted at 2026-08-08 04:03:49 on dist-test-slave-1l3l"
server_key: "f66a63a488707ff6cd6abb56e023567c"
server_key_iv: "1aed54f1593f36dc363948bb00f57ee4"
server_key_version: "encryptionkey@0"
I20260808 04:03:49.547083 5801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/minicluster-data/ts-1/wal/instance:
uuid: "c48da412664045f986366a3eb75b116a"
format_stamp: "Formatted at 2026-08-08 04:03:49 on dist-test-slave-1l3l"
server_key: "f66a63a488707ff6cd6abb56e023567c"
server_key_iv: "1aed54f1593f36dc363948bb00f57ee4"
server_key_version: "encryptionkey@0"
I20260808 04:03:49.551978 5801 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260808 04:03:49.555207 5820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:03:49.556592 5801 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:03:49.556748 5801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/minicluster-data/ts-1/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/minicluster-data/ts-1/wal
uuid: "c48da412664045f986366a3eb75b116a"
format_stamp: "Formatted at 2026-08-08 04:03:49 on dist-test-slave-1l3l"
server_key: "f66a63a488707ff6cd6abb56e023567c"
server_key_iv: "1aed54f1593f36dc363948bb00f57ee4"
server_key_version: "encryptionkey@0"
I20260808 04:03:49.556932 5801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:03:49.568033 5801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:03:49.571552 5801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:03:49.571849 5801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:03:49.572714 5801 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:03:49.573941 5801 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:03:49.574088 5801 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:03:49.574254 5801 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:03:49.574364 5801 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:03:49.605742 5801 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.2:33121
I20260808 04:03:49.605787 5961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.2:33121 every 8 connection(s)
I20260808 04:03:49.606963 5801 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
I20260808 04:03:49.613173 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 5801
I20260808 04:03:49.613379 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/minicluster-data/ts-1/wal/instance
I20260808 04:03:49.613930 5452 external_mini_cluster.cc:1428] Setting key dc40498ea25a55dce740917cca097c56
Aug 08 04:03:49 dist-test-slave-1l3l krb5kdc[5459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161829, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.2@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:03:49.627457 5795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:49.609620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.2:41519 (local address 127.5.83.62:41109)
0808 04:03:49.609821 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:49.609828 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:49.610988 (+ 1160us) server_negotiation.cc:407] Connection header received
0808 04:03:49.612794 (+ 1806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:49.612799 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:49.612883 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:49.612974 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:49.614890 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:49.615783 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:49.617316 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:49.617690 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:49.617798 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:49.621377 (+ 3579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:49.621384 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:49.621393 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:03:49.621432 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:49.624508 (+ 3076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:49.625193 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:49.625198 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:49.625201 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:49.625350 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:49.625761 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:49.625765 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:49.625768 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:49.626099 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:49.626234 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:49.626907 (+ 673us) server_negotiation.cc:299] Negotiation successful
0808 04:03:49.627126 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 04:03:49.628393 5964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:49.610142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:41109 (local address 127.5.83.2:41519)
0808 04:03:49.610825 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:49.610865 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 04:03:49.612545 (+ 1680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:03:49.613249 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:03:49.613260 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:03:49.613976 (+ 716us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:03:49.614703 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:49.614724 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:49.615939 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:49.615944 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:49.617158 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:49.617176 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:49.617914 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:49.617922 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:49.618135 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:49.618808 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:03:49.618832 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:03:49.621157 (+ 2325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:03:49.624636 (+ 3479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:03:49.624643 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:03:49.624654 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:03:49.625077 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:03:49.625445 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:03:49.625448 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:03:49.625451 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:03:49.625658 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:03:49.626208 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:03:49.626214 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:03:49.626545 (+ 331us) client_negotiation.cc:769] Sending connection context
0808 04:03:49.626780 (+ 235us) client_negotiation.cc:240] Negotiation successful
0808 04:03:49.627186 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":275,"threads_started":1}
I20260808 04:03:49.629922 5962 heartbeater.cc:347] Connected to a master server at 127.5.83.62:41109
I20260808 04:03:49.630218 5962 heartbeater.cc:464] Registering TS with master...
I20260808 04:03:49.630887 5962 heartbeater.cc:511] Master 127.5.83.62:41109 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.5.83.3@KRBTEST.COM; defaulting to no policy
I20260808 04:03:49.632316 5522 ts_manager.cc:194] Registered new tserver with Master: c48da412664045f986366a3eb75b116a (127.5.83.2:33121)
I20260808 04:03:49.633140 5522 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.2@KRBTEST.COM'} at 127.5.83.2:41519
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.3.keytab.
Entry for principal kudu/127.5.83.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.3.keytab.
I20260808 04:03:49.654017 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.3:0
--local_ip_for_outbound_sockets=127.5.83.3
--webserver_interface=127.5.83.3
--webserver_port=0
--tserver_master_addrs=127.5.83.62:41109
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.3.keytab
--principal=kudu/127.5.83.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:37819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:03:49.969470 5970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:03:49.969731 5970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:03:49.969833 5970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:03:49.976747 5970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:03:49.976841 5970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:03:49.976994 5970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.3
I20260808 04:03:49.986236 5970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:37819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.3.keytab
--principal=kudu/127.5.83.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.83.3
--webserver_port=0
--tserver_master_addrs=127.5.83.62:41109
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.3
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:03:49.987423 5970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:03:49.988679 5970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:03:50.001006 5977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:03:50.001051 5978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:03:50.001078 5980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:03:50.001744 5970 server_base.cc:1034] running on GCE node
I20260808 04:03:50.002315 5970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:03:50.003171 5970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:03:50.004406 5970 hybrid_clock.cc:648] HybridClock initialized: now 1786161830004349 us; error 34 us; skew 500 ppm
I20260808 04:03:50.004698 5970 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:03:50 dist-test-slave-1l3l krb5kdc[5459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161830, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:03:50.007637 5970 init.cc:377] Logged in from keytab as kudu/127.5.83.3@KRBTEST.COM (short username kudu)
I20260808 04:03:50.009023 5970 webserver.cc:460] Webserver started at http://127.5.83.3:46247/ using document root <none> and password file <none>
I20260808 04:03:50.009626 5970 fs_manager.cc:359] Metadata directory not provided
I20260808 04:03:50.009725 5970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:03:50.009969 5970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:03:50.012708 5970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/minicluster-data/ts-2/data/instance:
uuid: "bcfd859fcfb241388ac5527e4f827544"
format_stamp: "Formatted at 2026-08-08 04:03:50 on dist-test-slave-1l3l"
server_key: "c7fc0d240ad33a86210b90d4661b4171"
server_key_iv: "ca6ed3221a0a15f61a3ae712c6ebe9b6"
server_key_version: "encryptionkey@0"
I20260808 04:03:50.013374 5970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/minicluster-data/ts-2/wal/instance:
uuid: "bcfd859fcfb241388ac5527e4f827544"
format_stamp: "Formatted at 2026-08-08 04:03:50 on dist-test-slave-1l3l"
server_key: "c7fc0d240ad33a86210b90d4661b4171"
server_key_iv: "ca6ed3221a0a15f61a3ae712c6ebe9b6"
server_key_version: "encryptionkey@0"
I20260808 04:03:50.018168 5970 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260808 04:03:50.021313 5987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:03:50.022619 5970 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 04:03:50.022771 5970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/minicluster-data/ts-2/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/minicluster-data/ts-2/wal
uuid: "bcfd859fcfb241388ac5527e4f827544"
format_stamp: "Formatted at 2026-08-08 04:03:50 on dist-test-slave-1l3l"
server_key: "c7fc0d240ad33a86210b90d4661b4171"
server_key_iv: "ca6ed3221a0a15f61a3ae712c6ebe9b6"
server_key_version: "encryptionkey@0"
I20260808 04:03:50.022956 5970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:03:50.038305 5970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:03:50.042001 5970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:03:50.042253 5970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:03:50.043135 5970 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:03:50.044492 5970 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:03:50.044565 5970 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:03:50.044679 5970 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:03:50.044739 5970 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:03:50.077368 5970 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.3:46635
I20260808 04:03:50.077436 6135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.3:46635 every 8 connection(s)
I20260808 04:03:50.078683 5970 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
I20260808 04:03:50.088819 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 5970
I20260808 04:03:50.089007 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161826605655-5452-0/minicluster-data/ts-2/wal/instance
I20260808 04:03:50.089591 5452 external_mini_cluster.cc:1428] Setting key edd6270e20f910ac0b21bafe4c316b5b
Aug 08 04:03:50 dist-test-slave-1l3l krb5kdc[5459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161830, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.3@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:03:50.098258 5795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:50.081397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.3:46859 (local address 127.5.83.62:41109)
0808 04:03:50.081588 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:50.081592 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:50.082725 (+ 1133us) server_negotiation.cc:407] Connection header received
0808 04:03:50.084552 (+ 1827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:50.084558 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:50.084634 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:50.084721 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:50.086128 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:50.086868 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:50.088548 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:50.088942 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:50.089059 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:50.092807 (+ 3748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:50.092811 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:50.092815 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:03:50.092846 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:50.095553 (+ 2707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:50.096269 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:50.096273 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:50.096275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:50.096410 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:50.096752 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:50.096755 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:50.096757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:50.097070 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:50.097174 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:50.097852 (+ 678us) server_negotiation.cc:299] Negotiation successful
0808 04:03:50.098020 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260808 04:03:50.099238 6138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:50.081851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:41109 (local address 127.5.83.3:46859)
0808 04:03:50.082568 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:50.082609 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 04:03:50.084295 (+ 1686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:03:50.084957 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:03:50.084965 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:03:50.085439 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:03:50.085968 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:50.085983 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:50.087037 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:50.087043 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:50.088413 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:50.088425 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:50.089169 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:50.089177 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:50.089466 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:50.090127 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:03:50.090157 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:03:50.092616 (+ 2459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:03:50.095667 (+ 3051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:03:50.095674 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:03:50.095684 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:03:50.096172 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:03:50.096487 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:03:50.096490 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:03:50.096492 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:03:50.096683 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:03:50.097171 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:03:50.097177 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:03:50.097481 (+ 304us) client_negotiation.cc:769] Sending connection context
0808 04:03:50.097762 (+ 281us) client_negotiation.cc:240] Negotiation successful
0808 04:03:50.098167 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":290,"threads_started":1}
I20260808 04:03:50.100469 6136 heartbeater.cc:347] Connected to a master server at 127.5.83.62:41109
I20260808 04:03:50.100736 6136 heartbeater.cc:464] Registering TS with master...
I20260808 04:03:50.101414 6136 heartbeater.cc:511] Master 127.5.83.62:41109 requested a full tablet report, sending...
I20260808 04:03:50.102706 5522 ts_manager.cc:194] Registered new tserver with Master: bcfd859fcfb241388ac5527e4f827544 (127.5.83.3:46635)
I20260808 04:03:50.103470 5522 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.3@KRBTEST.COM'} at 127.5.83.3:46859
I20260808 04:03:50.106082 5452 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 04:03:50 dist-test-slave-1l3l krb5kdc[5459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161830, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 04:03:50 dist-test-slave-1l3l krb5kdc[5459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161830, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.83.1@KRBTEST.COM
I20260808 04:03:50.161540 6144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:50.146690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45062 (local address 127.5.83.1:42913)
0808 04:03:50.147316 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:50.147325 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:50.147432 (+ 107us) server_negotiation.cc:407] Connection header received
0808 04:03:50.147626 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:50.147633 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:50.147852 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:50.147996 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:50.149427 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:50.150408 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:50.151688 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:50.152275 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:50.152401 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:50.156094 (+ 3693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:50.156105 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:50.156117 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:03:50.156147 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:50.158773 (+ 2626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:50.159393 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:50.159399 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:50.159401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:50.159520 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:50.159964 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:50.159968 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:50.159970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:50.160362 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:50.160631 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:50.160918 (+ 287us) server_negotiation.cc:299] Negotiation successful
0808 04:03:50.161167 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":300,"threads_started":1}
W20260808 04:03:50.162949 5672 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45062
I20260808 04:03:50.176983 5793 heartbeater.cc:503] Master 127.5.83.62:41109 was elected leader, sending a full tablet report...
Aug 08 04:03:50 dist-test-slave-1l3l krb5kdc[5459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 04:03:50 dist-test-slave-1l3l krb5kdc[5459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.1@KRBTEST.COM
I20260808 04:03:50.210109 6144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:50.195842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45074 (local address 127.5.83.1:42913)
0808 04:03:50.196019 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:50.196026 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:50.196367 (+ 341us) server_negotiation.cc:407] Connection header received
0808 04:03:50.196528 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:50.196532 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:50.196593 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:50.196675 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:50.198016 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:50.198708 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:50.199899 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:50.200309 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:50.200427 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:50.203941 (+ 3514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:50.203946 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:50.203949 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:03:50.203980 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:50.206932 (+ 2952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:50.207785 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:50.207790 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:50.207794 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:50.207966 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:50.208408 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:50.208414 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:50.208417 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:50.208741 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:50.208988 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:50.209435 (+ 447us) server_negotiation.cc:299] Negotiation successful
0808 04:03:50.209731 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 04:03:50.212146 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 5608
I20260808 04:03:50.226576 6144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:50.213629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45082 (local address 127.5.83.1:42913)
0808 04:03:50.213786 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:50.213791 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:50.215420 (+ 1629us) server_negotiation.cc:407] Connection header received
0808 04:03:50.215552 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:50.215557 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:50.215626 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:50.215751 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:50.217107 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:50.217852 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:50.219025 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:50.219427 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:50.219519 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:50.221828 (+ 2309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:50.221833 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:50.221836 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:03:50.221865 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:50.224349 (+ 2484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:50.224998 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:50.225001 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:50.225004 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:50.225124 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:50.225443 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:50.225446 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:50.225448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:50.225670 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:50.225871 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:50.226103 (+ 232us) server_negotiation.cc:299] Negotiation successful
0808 04:03:50.226243 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 04:03:50.227342 5785 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:03:50.312796 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 5608
I20260808 04:03:50.331768 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 5801
Aug 08 04:03:50 dist-test-slave-1l3l krb5kdc[5459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.2@KRBTEST.COM
I20260808 04:03:50.346619 6154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:50.332711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44698 (local address 127.5.83.2:33121)
0808 04:03:50.333218 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:50.333225 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:50.333250 (+ 25us) server_negotiation.cc:407] Connection header received
0808 04:03:50.333307 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:50.333313 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:50.333514 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:50.333632 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:50.334742 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:50.335810 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:50.336969 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:50.337488 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:50.337583 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:50.341705 (+ 4122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:50.341715 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:50.341728 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:03:50.341759 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:50.344373 (+ 2614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:50.344915 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:50.344921 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:50.344923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:50.345042 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:50.345319 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:50.345322 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:50.345324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:50.345639 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:50.345864 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:50.346090 (+ 226us) server_negotiation.cc:299] Negotiation successful
0808 04:03:50.346274 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":243,"threads_started":1}
I20260808 04:03:50.347874 5955 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:03:50.429078 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 5801
I20260808 04:03:50.447817 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 5970
Aug 08 04:03:50 dist-test-slave-1l3l krb5kdc[5459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.3@KRBTEST.COM
I20260808 04:03:50.463408 6158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:50.448874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57884 (local address 127.5.83.3:46635)
0808 04:03:50.449533 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:50.449540 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:50.449571 (+ 31us) server_negotiation.cc:407] Connection header received
0808 04:03:50.449629 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:50.449635 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:50.449849 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:50.449991 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:50.451181 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:50.452293 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:50.453750 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:50.454299 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:50.454394 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:50.458572 (+ 4178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:50.458582 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:50.458595 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:03:50.458627 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:50.461127 (+ 2500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:50.461762 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:50.461768 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:50.461770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:50.461880 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:50.462157 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:50.462160 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:50.462162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:50.462476 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:50.462696 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:50.462870 (+ 174us) server_negotiation.cc:299] Negotiation successful
0808 04:03:50.463049 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":353,"threads_started":1}
I20260808 04:03:50.464648 6131 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:03:50.548998 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 5970
I20260808 04:03:50.567696 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 5479
I20260808 04:03:50.568922 5569 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:03:50.670944 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 5479
2026-08-08T04:03:50Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4009 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:03:50 dist-test-slave-1l3l krb5kdc[6164](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:03:50 dist-test-slave-1l3l krb5kdc[6164](info): set up 2 sockets
Aug 08 04:03:50 dist-test-slave-1l3l krb5kdc[6164](info): commencing operation
krb5kdc: starting...
W20260808 04:03:51.827255 5452 mini_kdc.cc:121] Time spent starting KDC: real 1.112s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/krb5kdc/test-user.keytab.
Aug 08 04:03:51 dist-test-slave-1l3l krb5kdc[6164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:03:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:03:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.83.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab.
Entry for principal kudu/127.5.83.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab.
I20260808 04:03:51.998998 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:45043
--webserver_interface=127.5.83.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab
--principal=kudu/127.5.83.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:36271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.62:45043
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:03:52.309656 6180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:03:52.309924 6180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:03:52.310019 6180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:03:52.317116 6180 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:03:52.317219 6180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:03:52.317269 6180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:03:52.317330 6180 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:03:52.317373 6180 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:03:52.326807 6180 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:36271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.62:45043
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab
--principal=kudu/127.5.83.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:45043
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:03:52.328050 6180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:03:52.329309 6180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:03:52.340237 6185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:03:52.340317 6186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:03:52.340876 6191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:03:52.341154 6180 server_base.cc:1034] running on GCE node
I20260808 04:03:52.341903 6180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:03:52.343209 6180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:03:52.344421 6180 hybrid_clock.cc:648] HybridClock initialized: now 1786161832344360 us; error 40 us; skew 500 ppm
I20260808 04:03:52.344735 6180 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:03:52 dist-test-slave-1l3l krb5kdc[6164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161832, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:03:52.350490 6180 init.cc:377] Logged in from keytab as kudu/127.5.83.62@KRBTEST.COM (short username kudu)
I20260808 04:03:52.352078 6180 webserver.cc:460] Webserver started at http://127.5.83.62:44283/ using document root <none> and password file <none>
I20260808 04:03:52.352741 6180 fs_manager.cc:359] Metadata directory not provided
I20260808 04:03:52.352856 6180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:03:52.353166 6180 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:03:52.356112 6180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/minicluster-data/master-0/data/instance:
uuid: "6beae8e6013448ddb4b85340ae776738"
format_stamp: "Formatted at 2026-08-08 04:03:52 on dist-test-slave-1l3l"
server_key: "a4531b85702d1ab62daa8bcabf9b11e1"
server_key_iv: "5d2e5cc8438385d05f6f98392df1f5b9"
server_key_version: "encryptionkey@0"
I20260808 04:03:52.356853 6180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/minicluster-data/master-0/wal/instance:
uuid: "6beae8e6013448ddb4b85340ae776738"
format_stamp: "Formatted at 2026-08-08 04:03:52 on dist-test-slave-1l3l"
server_key: "a4531b85702d1ab62daa8bcabf9b11e1"
server_key_iv: "5d2e5cc8438385d05f6f98392df1f5b9"
server_key_version: "encryptionkey@0"
I20260808 04:03:52.362013 6180 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260808 04:03:52.365612 6199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:03:52.366892 6180 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 04:03:52.367049 6180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/minicluster-data/master-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/minicluster-data/master-0/wal
uuid: "6beae8e6013448ddb4b85340ae776738"
format_stamp: "Formatted at 2026-08-08 04:03:52 on dist-test-slave-1l3l"
server_key: "a4531b85702d1ab62daa8bcabf9b11e1"
server_key_iv: "5d2e5cc8438385d05f6f98392df1f5b9"
server_key_version: "encryptionkey@0"
I20260808 04:03:52.367228 6180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:03:52.390123 6180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:03:52.394161 6180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:03:52.394430 6180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:03:52.411628 6180 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.62:45043
I20260808 04:03:52.411664 6273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.62:45043 every 8 connection(s)
I20260808 04:03:52.412982 6180 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
I20260808 04:03:52.416595 6274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:03:52.423283 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 6180
I20260808 04:03:52.423517 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/minicluster-data/master-0/wal/instance
I20260808 04:03:52.424326 5452 external_mini_cluster.cc:1428] Setting key 8e7931af5a07309c0780a1e095b13bcb
I20260808 04:03:52.427053 6274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6beae8e6013448ddb4b85340ae776738: Bootstrap starting.
I20260808 04:03:52.430154 6274 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6beae8e6013448ddb4b85340ae776738: Neither blocks nor log segments found. Creating new log.
I20260808 04:03:52.431371 6274 log.cc:824] T 00000000000000000000000000000000 P 6beae8e6013448ddb4b85340ae776738: Log is configured to *not* fsync() on all Append() calls
I20260808 04:03:52.434345 6274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6beae8e6013448ddb4b85340ae776738: No bootstrap required, opened a new log
Aug 08 04:03:52 dist-test-slave-1l3l krb5kdc[6164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:03:52.442770 6274 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6beae8e6013448ddb4b85340ae776738 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6beae8e6013448ddb4b85340ae776738" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 45043 } }
I20260808 04:03:52.443145 6274 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6beae8e6013448ddb4b85340ae776738 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:03:52.443245 6274 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6beae8e6013448ddb4b85340ae776738 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6beae8e6013448ddb4b85340ae776738, State: Initialized, Role: FOLLOWER
I20260808 04:03:52.443959 6274 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6beae8e6013448ddb4b85340ae776738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6beae8e6013448ddb4b85340ae776738" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 45043 } }
I20260808 04:03:52.444211 6274 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6beae8e6013448ddb4b85340ae776738 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:03:52.444365 6274 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6beae8e6013448ddb4b85340ae776738 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:03:52.444518 6274 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6beae8e6013448ddb4b85340ae776738 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:03:52.447815 6277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:52.426291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52242 (local address 127.5.83.62:45043)
0808 04:03:52.427023 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:52.427035 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:52.427082 (+ 47us) server_negotiation.cc:407] Connection header received
0808 04:03:52.428844 (+ 1762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:52.428866 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:52.429206 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:52.429497 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:52.431247 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:52.432931 (+ 1684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:52.434624 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:52.435261 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:52.435416 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:52.440090 (+ 4674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:52.440101 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:52.440115 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:03:52.440150 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:52.443162 (+ 3012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:52.443958 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:52.443964 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:52.443970 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:52.444158 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:52.444598 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:52.444604 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:52.444607 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:52.445291 (+ 684us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:52.445635 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:52.445995 (+ 360us) server_negotiation.cc:299] Negotiation successful
0808 04:03:52.446446 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":345,"threads_started":1}
I20260808 04:03:52.447844 6274 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6beae8e6013448ddb4b85340ae776738 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6beae8e6013448ddb4b85340ae776738" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 45043 } }
I20260808 04:03:52.448427 6274 leader_election.cc:302] T 00000000000000000000000000000000 P 6beae8e6013448ddb4b85340ae776738 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6beae8e6013448ddb4b85340ae776738; no voters:
I20260808 04:03:52.449102 6274 leader_election.cc:288] T 00000000000000000000000000000000 P 6beae8e6013448ddb4b85340ae776738 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:03:52.449198 6279 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6beae8e6013448ddb4b85340ae776738 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:03:52.450400 6279 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6beae8e6013448ddb4b85340ae776738 [term 1 LEADER]: Becoming Leader. State: Replica: 6beae8e6013448ddb4b85340ae776738, State: Running, Role: LEADER
I20260808 04:03:52.450975 6279 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6beae8e6013448ddb4b85340ae776738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6beae8e6013448ddb4b85340ae776738" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 45043 } }
I20260808 04:03:52.451414 6274 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6beae8e6013448ddb4b85340ae776738 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:03:52.453934 6280 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6beae8e6013448ddb4b85340ae776738 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6beae8e6013448ddb4b85340ae776738" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6beae8e6013448ddb4b85340ae776738" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 45043 } } }
I20260808 04:03:52.454276 6280 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6beae8e6013448ddb4b85340ae776738 [sys.catalog]: This master's current role is: LEADER
I20260808 04:03:52.455937 6281 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6beae8e6013448ddb4b85340ae776738 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6beae8e6013448ddb4b85340ae776738. Latest consensus state: current_term: 1 leader_uuid: "6beae8e6013448ddb4b85340ae776738" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6beae8e6013448ddb4b85340ae776738" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 45043 } } }
I20260808 04:03:52.456248 6281 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6beae8e6013448ddb4b85340ae776738 [sys.catalog]: This master's current role is: LEADER
I20260808 04:03:52.456875 6288 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:03:52.462160 6288 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:03:52.471995 6288 catalog_manager.cc:1347] Generated new cluster ID: 7b2b168c0efd4c70bb51f3a3d03214ab
I20260808 04:03:52.472091 6288 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:03:52.499748 6288 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:03:52.501027 6288 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:03:52.507630 6288 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6beae8e6013448ddb4b85340ae776738: Generated new TSK 0
I20260808 04:03:52.508384 6288 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.83.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.1.keytab.
Entry for principal kudu/127.5.83.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.1.keytab.
I20260808 04:03:52.559065 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.1:0
--local_ip_for_outbound_sockets=127.5.83.1
--webserver_interface=127.5.83.1
--webserver_port=0
--tserver_master_addrs=127.5.83.62:45043
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.1.keytab
--principal=kudu/127.5.83.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:36271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:03:52.900686 6309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:03:52.900938 6309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:03:52.901041 6309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:03:52.907837 6309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:03:52.907924 6309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:03:52.908036 6309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.1
I20260808 04:03:52.917142 6309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:36271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.1.keytab
--principal=kudu/127.5.83.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.83.1
--webserver_port=0
--tserver_master_addrs=127.5.83.62:45043
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.1
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:03:52.918218 6309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:03:52.919411 6309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:03:52.931228 6315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:03:52.931293 6319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:03:52.931270 6317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:03:52.932024 6309 server_base.cc:1034] running on GCE node
I20260808 04:03:52.932591 6309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:03:52.933499 6309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:03:52.934695 6309 hybrid_clock.cc:648] HybridClock initialized: now 1786161832934644 us; error 45 us; skew 500 ppm
I20260808 04:03:52.934999 6309 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:03:52 dist-test-slave-1l3l krb5kdc[6164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161832, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:03:52.938097 6309 init.cc:377] Logged in from keytab as kudu/127.5.83.1@KRBTEST.COM (short username kudu)
I20260808 04:03:52.939363 6309 webserver.cc:460] Webserver started at http://127.5.83.1:35209/ using document root <none> and password file <none>
I20260808 04:03:52.940019 6309 fs_manager.cc:359] Metadata directory not provided
I20260808 04:03:52.940119 6309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:03:52.940366 6309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:03:52.943158 6309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/minicluster-data/ts-0/data/instance:
uuid: "360e3415fb794cfba2f30ada9f50fa7a"
format_stamp: "Formatted at 2026-08-08 04:03:52 on dist-test-slave-1l3l"
server_key: "0a48a16f68fb7a155aab76bd2c43ab3e"
server_key_iv: "1dd11e44c4f69a6f2d3265c13ea8a4ac"
server_key_version: "encryptionkey@0"
I20260808 04:03:52.943928 6309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/minicluster-data/ts-0/wal/instance:
uuid: "360e3415fb794cfba2f30ada9f50fa7a"
format_stamp: "Formatted at 2026-08-08 04:03:52 on dist-test-slave-1l3l"
server_key: "0a48a16f68fb7a155aab76bd2c43ab3e"
server_key_iv: "1dd11e44c4f69a6f2d3265c13ea8a4ac"
server_key_version: "encryptionkey@0"
I20260808 04:03:52.948930 6309 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260808 04:03:52.952128 6327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:03:52.953351 6309 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 04:03:52.953495 6309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/minicluster-data/ts-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/minicluster-data/ts-0/wal
uuid: "360e3415fb794cfba2f30ada9f50fa7a"
format_stamp: "Formatted at 2026-08-08 04:03:52 on dist-test-slave-1l3l"
server_key: "0a48a16f68fb7a155aab76bd2c43ab3e"
server_key_iv: "1dd11e44c4f69a6f2d3265c13ea8a4ac"
server_key_version: "encryptionkey@0"
I20260808 04:03:52.953646 6309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:03:52.966293 6309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:03:52.969726 6309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:03:52.969950 6309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:03:52.970820 6309 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:03:52.972179 6309 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:03:52.972251 6309 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:03:52.972335 6309 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:03:52.972386 6309 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:03:53.002131 6309 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.1:40423
I20260808 04:03:53.002169 6483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.1:40423 every 8 connection(s)
I20260808 04:03:53.003309 6309 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
I20260808 04:03:53.006878 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 6309
I20260808 04:03:53.007103 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/minicluster-data/ts-0/wal/instance
I20260808 04:03:53.007949 5452 external_mini_cluster.cc:1428] Setting key 20628b4542d1503f70815c9706698114
Aug 08 04:03:53 dist-test-slave-1l3l krb5kdc[6164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161832, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.1@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:03:53.023252 6487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:53.005717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.1:34003 (local address 127.5.83.62:45043)
0808 04:03:53.006162 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:53.006166 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:53.007252 (+ 1086us) server_negotiation.cc:407] Connection header received
0808 04:03:53.009621 (+ 2369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:53.009625 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:53.009689 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:53.009799 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:53.011309 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:53.012341 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:53.013838 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:53.014340 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:53.014468 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:53.018024 (+ 3556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:53.018028 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:53.018031 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:03:53.018062 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:53.020449 (+ 2387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:53.021214 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:53.021217 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:53.021219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:53.021321 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:53.021730 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:53.021733 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:53.021735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:53.021966 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:53.022077 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:53.022853 (+ 776us) server_negotiation.cc:299] Negotiation successful
0808 04:03:53.023008 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":228,"threads_started":1}
I20260808 04:03:53.024749 6488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:53.006219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:45043 (local address 127.5.83.1:34003)
0808 04:03:53.007042 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:53.007107 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0808 04:03:53.009309 (+ 2202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:03:53.010084 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:03:53.010096 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:03:53.010594 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:03:53.011138 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:53.011154 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:53.012491 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:53.012495 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:53.013699 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:53.013711 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:53.014561 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:53.014568 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:53.014798 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:53.015487 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:03:53.015516 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:03:53.017817 (+ 2301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:03:53.020582 (+ 2765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:03:53.020591 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:03:53.020604 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:03:53.021123 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:03:53.021410 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:03:53.021415 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:03:53.021418 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:03:53.021642 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:03:53.022086 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:03:53.022093 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:03:53.022476 (+ 383us) client_negotiation.cc:769] Sending connection context
0808 04:03:53.022779 (+ 303us) client_negotiation.cc:240] Negotiation successful
0808 04:03:53.023284 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":322,"threads_started":1}
WARNING: no policy specified for kudu/127.5.83.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.2@KRBTEST.COM" created.
I20260808 04:03:53.033105 6484 heartbeater.cc:347] Connected to a master server at 127.5.83.62:45043
I20260808 04:03:53.033468 6484 heartbeater.cc:464] Registering TS with master...
I20260808 04:03:53.034312 6484 heartbeater.cc:511] Master 127.5.83.62:45043 requested a full tablet report, sending...
I20260808 04:03:53.036767 6224 ts_manager.cc:194] Registered new tserver with Master: 360e3415fb794cfba2f30ada9f50fa7a (127.5.83.1:40423)
I20260808 04:03:53.038795 6224 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.1@KRBTEST.COM'} at 127.5.83.1:34003
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.2.keytab.
Entry for principal kudu/127.5.83.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.2.keytab.
I20260808 04:03:53.056802 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.2:0
--local_ip_for_outbound_sockets=127.5.83.2
--webserver_interface=127.5.83.2
--webserver_port=0
--tserver_master_addrs=127.5.83.62:45043
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.2.keytab
--principal=kudu/127.5.83.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:36271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:03:53.366863 6491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:03:53.367128 6491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:03:53.367223 6491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:03:53.374318 6491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:03:53.374418 6491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:03:53.374539 6491 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.2
I20260808 04:03:53.383754 6491 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:36271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.2.keytab
--principal=kudu/127.5.83.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.83.2
--webserver_port=0
--tserver_master_addrs=127.5.83.62:45043
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.2
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:03:53.384975 6491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:03:53.386250 6491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:03:53.400348 6498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:03:53.400629 6497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:03:53.401352 6500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:03:53.401757 6491 server_base.cc:1034] running on GCE node
I20260808 04:03:53.402307 6491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:03:53.403199 6491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:03:53.404395 6491 hybrid_clock.cc:648] HybridClock initialized: now 1786161833404341 us; error 41 us; skew 500 ppm
I20260808 04:03:53.404685 6491 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:03:53 dist-test-slave-1l3l krb5kdc[6164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161833, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:03:53.407631 6491 init.cc:377] Logged in from keytab as kudu/127.5.83.2@KRBTEST.COM (short username kudu)
I20260808 04:03:53.408980 6491 webserver.cc:460] Webserver started at http://127.5.83.2:34745/ using document root <none> and password file <none>
I20260808 04:03:53.409595 6491 fs_manager.cc:359] Metadata directory not provided
I20260808 04:03:53.409709 6491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:03:53.410007 6491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:03:53.412844 6491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/minicluster-data/ts-1/data/instance:
uuid: "f21c4e80bd0d46919e76b7237ac9c714"
format_stamp: "Formatted at 2026-08-08 04:03:53 on dist-test-slave-1l3l"
server_key: "fb163631da7e1a47c0615dd6ff1af019"
server_key_iv: "cd68a9489d78db1270a8450b72e79133"
server_key_version: "encryptionkey@0"
I20260808 04:03:53.413599 6491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/minicluster-data/ts-1/wal/instance:
uuid: "f21c4e80bd0d46919e76b7237ac9c714"
format_stamp: "Formatted at 2026-08-08 04:03:53 on dist-test-slave-1l3l"
server_key: "fb163631da7e1a47c0615dd6ff1af019"
server_key_iv: "cd68a9489d78db1270a8450b72e79133"
server_key_version: "encryptionkey@0"
I20260808 04:03:53.418476 6491 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 04:03:53.421766 6508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:03:53.423069 6491 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 04:03:53.423209 6491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/minicluster-data/ts-1/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/minicluster-data/ts-1/wal
uuid: "f21c4e80bd0d46919e76b7237ac9c714"
format_stamp: "Formatted at 2026-08-08 04:03:53 on dist-test-slave-1l3l"
server_key: "fb163631da7e1a47c0615dd6ff1af019"
server_key_iv: "cd68a9489d78db1270a8450b72e79133"
server_key_version: "encryptionkey@0"
I20260808 04:03:53.423374 6491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:03:53.461710 6491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:03:53.465834 6491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:03:53.466104 6491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:03:53.467043 6491 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:03:53.468355 6491 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:03:53.468449 6491 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:03:53.468569 6491 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:03:53.468626 6491 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:03:53.498384 6491 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.2:45547
I20260808 04:03:53.498457 6666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.2:45547 every 8 connection(s)
I20260808 04:03:53.499584 6491 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
I20260808 04:03:53.501627 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 6491
I20260808 04:03:53.501834 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/minicluster-data/ts-1/wal/instance
I20260808 04:03:53.502430 5452 external_mini_cluster.cc:1428] Setting key d13c1c1bf054306dea4b77fcd530da33
Aug 08 04:03:53 dist-test-slave-1l3l krb5kdc[6164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161833, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.2@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:03:53.518512 6487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:53.502078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.2:47051 (local address 127.5.83.62:45043)
0808 04:03:53.502236 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:53.502241 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:53.503688 (+ 1447us) server_negotiation.cc:407] Connection header received
0808 04:03:53.505714 (+ 2026us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:53.505719 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:53.505797 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:53.505880 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:53.507298 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:53.508080 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:53.509550 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:53.509954 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:53.510071 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:53.513306 (+ 3235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:53.513311 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:53.513314 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:03:53.513344 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:53.515767 (+ 2423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:53.516415 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:53.516420 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:53.516424 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:53.516548 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:53.516934 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:53.516939 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:53.516943 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:53.517192 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:53.517348 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:53.517967 (+ 619us) server_negotiation.cc:299] Negotiation successful
0808 04:03:53.518181 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
WARNING: no policy specified for kudu/127.5.83.3@KRBTEST.COM; defaulting to no policy
I20260808 04:03:53.519423 6670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:53.502720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:45043 (local address 127.5.83.2:47051)
0808 04:03:53.503526 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:53.503575 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0808 04:03:53.505469 (+ 1894us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:03:53.506111 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:03:53.506119 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:03:53.506605 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:03:53.507138 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:53.507159 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:53.508224 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:53.508228 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:53.509415 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:53.509427 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:53.510170 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:53.510177 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:53.510390 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:53.511047 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:03:53.511068 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:03:53.513099 (+ 2031us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:03:53.515882 (+ 2783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:03:53.515891 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:03:53.515905 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:03:53.516305 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:03:53.516651 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:03:53.516656 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:03:53.516660 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:03:53.516828 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:03:53.517314 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:03:53.517321 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:03:53.517624 (+ 303us) client_negotiation.cc:769] Sending connection context
0808 04:03:53.517859 (+ 235us) client_negotiation.cc:240] Negotiation successful
0808 04:03:53.518324 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":345,"threads_started":1}
I20260808 04:03:53.520736 6667 heartbeater.cc:347] Connected to a master server at 127.5.83.62:45043
I20260808 04:03:53.521040 6667 heartbeater.cc:464] Registering TS with master...
I20260808 04:03:53.521714 6667 heartbeater.cc:511] Master 127.5.83.62:45043 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.3@KRBTEST.COM" created.
I20260808 04:03:53.523056 6224 ts_manager.cc:194] Registered new tserver with Master: f21c4e80bd0d46919e76b7237ac9c714 (127.5.83.2:45547)
I20260808 04:03:53.523874 6224 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.2@KRBTEST.COM'} at 127.5.83.2:47051
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.3.keytab.
Entry for principal kudu/127.5.83.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.3.keytab.
I20260808 04:03:53.540934 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.3:0
--local_ip_for_outbound_sockets=127.5.83.3
--webserver_interface=127.5.83.3
--webserver_port=0
--tserver_master_addrs=127.5.83.62:45043
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.3.keytab
--principal=kudu/127.5.83.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:36271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:03:53.848976 6673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:03:53.849242 6673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:03:53.849350 6673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:03:53.856303 6673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:03:53.856402 6673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:03:53.856520 6673 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.3
I20260808 04:03:53.865986 6673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:36271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.3.keytab
--principal=kudu/127.5.83.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.83.3
--webserver_port=0
--tserver_master_addrs=127.5.83.62:45043
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.3
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:03:53.867094 6673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:03:53.868340 6673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:03:53.880591 6682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:03:53.880640 6685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:03:53.880666 6683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:03:53.881328 6673 server_base.cc:1034] running on GCE node
I20260808 04:03:53.881887 6673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:03:53.882717 6673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:03:53.883906 6673 hybrid_clock.cc:648] HybridClock initialized: now 1786161833883837 us; error 52 us; skew 500 ppm
I20260808 04:03:53.884182 6673 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:03:53 dist-test-slave-1l3l krb5kdc[6164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161833, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:03:53.887001 6673 init.cc:377] Logged in from keytab as kudu/127.5.83.3@KRBTEST.COM (short username kudu)
I20260808 04:03:53.888355 6673 webserver.cc:460] Webserver started at http://127.5.83.3:43487/ using document root <none> and password file <none>
I20260808 04:03:53.888978 6673 fs_manager.cc:359] Metadata directory not provided
I20260808 04:03:53.889096 6673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:03:53.889374 6673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:03:53.892119 6673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/minicluster-data/ts-2/data/instance:
uuid: "ac18d15cd5d143c78fb00dc74e5d8802"
format_stamp: "Formatted at 2026-08-08 04:03:53 on dist-test-slave-1l3l"
server_key: "17fd90fecd07433a2f29fcf004a34635"
server_key_iv: "b50e4336c8686f934a686e6821b9ffcf"
server_key_version: "encryptionkey@0"
I20260808 04:03:53.892849 6673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/minicluster-data/ts-2/wal/instance:
uuid: "ac18d15cd5d143c78fb00dc74e5d8802"
format_stamp: "Formatted at 2026-08-08 04:03:53 on dist-test-slave-1l3l"
server_key: "17fd90fecd07433a2f29fcf004a34635"
server_key_iv: "b50e4336c8686f934a686e6821b9ffcf"
server_key_version: "encryptionkey@0"
I20260808 04:03:53.897560 6673 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 04:03:53.900784 6694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:03:53.901995 6673 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260808 04:03:53.902151 6673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/minicluster-data/ts-2/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/minicluster-data/ts-2/wal
uuid: "ac18d15cd5d143c78fb00dc74e5d8802"
format_stamp: "Formatted at 2026-08-08 04:03:53 on dist-test-slave-1l3l"
server_key: "17fd90fecd07433a2f29fcf004a34635"
server_key_iv: "b50e4336c8686f934a686e6821b9ffcf"
server_key_version: "encryptionkey@0"
I20260808 04:03:53.902320 6673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:03:53.919399 6673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:03:53.923442 6673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:03:53.923736 6673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:03:53.924628 6673 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:03:53.925885 6673 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:03:53.925963 6673 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:03:53.926098 6673 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:03:53.926159 6673 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:03:53.956009 6673 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.3:33067
I20260808 04:03:53.956070 6863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.3:33067 every 8 connection(s)
I20260808 04:03:53.957199 6673 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
I20260808 04:03:53.965763 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 6673
I20260808 04:03:53.965930 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161826605655-5452-0/minicluster-data/ts-2/wal/instance
I20260808 04:03:53.966356 5452 external_mini_cluster.cc:1428] Setting key 3dd7bad4e72d69100503d6da2e896c1f
Aug 08 04:03:53 dist-test-slave-1l3l krb5kdc[6164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161833, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.3@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:03:53.976270 6487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:53.959697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.3:35053 (local address 127.5.83.62:45043)
0808 04:03:53.959917 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:53.959923 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:53.961141 (+ 1218us) server_negotiation.cc:407] Connection header received
0808 04:03:53.962928 (+ 1787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:53.962932 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:53.963001 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:53.963088 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:53.964762 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:53.965554 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:53.967208 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:53.967594 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:53.967733 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:53.970926 (+ 3193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:53.970932 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:53.970935 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:03:53.970967 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:53.973422 (+ 2455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:53.974125 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:53.974129 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:53.974132 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:53.974249 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:53.974645 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:53.974648 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:53.974651 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:53.974902 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:53.975043 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:53.975694 (+ 651us) server_negotiation.cc:299] Negotiation successful
0808 04:03:53.975937 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 04:03:53.977062 6866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:53.960227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:45043 (local address 127.5.83.3:35053)
0808 04:03:53.960958 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:53.960998 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 04:03:53.962688 (+ 1690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:03:53.963359 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:03:53.963367 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:03:53.964022 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:03:53.964596 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:53.964613 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:53.965712 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:53.965716 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:53.967038 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:53.967051 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:53.967842 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:53.967850 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:53.968068 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:53.968661 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:03:53.968682 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:03:53.970723 (+ 2041us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:03:53.973546 (+ 2823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:03:53.973554 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:03:53.973568 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:03:53.974005 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:03:53.974353 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:03:53.974358 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:03:53.974362 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:03:53.974534 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:03:53.975029 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:03:53.975037 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:03:53.975331 (+ 294us) client_negotiation.cc:769] Sending connection context
0808 04:03:53.975562 (+ 231us) client_negotiation.cc:240] Negotiation successful
0808 04:03:53.976065 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":282,"threads_started":1}
I20260808 04:03:53.978152 6864 heartbeater.cc:347] Connected to a master server at 127.5.83.62:45043
I20260808 04:03:53.978437 6864 heartbeater.cc:464] Registering TS with master...
I20260808 04:03:53.979094 6864 heartbeater.cc:511] Master 127.5.83.62:45043 requested a full tablet report, sending...
I20260808 04:03:53.980564 6224 ts_manager.cc:194] Registered new tserver with Master: ac18d15cd5d143c78fb00dc74e5d8802 (127.5.83.3:33067)
I20260808 04:03:53.981446 6224 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.3@KRBTEST.COM'} at 127.5.83.3:35053
I20260808 04:03:53.981568 5452 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:03:53.999948 6487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:53.988273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52258 (local address 127.5.83.62:45043)
0808 04:03:53.988451 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:53.988455 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:53.989503 (+ 1048us) server_negotiation.cc:407] Connection header received
0808 04:03:53.989618 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:53.989621 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:53.989677 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:53.989749 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:53.991134 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:53.991870 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:53.993112 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:53.993511 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:53.993601 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:53.995728 (+ 2127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:53.995735 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:53.995739 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:03:53.995773 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:53.997969 (+ 2196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:53.998525 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:53.998528 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:53.998531 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:53.998631 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:53.998878 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:53.998881 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:53.998883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:53.999143 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:53.999309 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:53.999542 (+ 233us) server_negotiation.cc:299] Negotiation successful
0808 04:03:53.999727 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260808 04:03:54.008975 6224 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52258:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:03:54.015971 6224 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:03:54.041951 6484 heartbeater.cc:503] Master 127.5.83.62:45043 was elected leader, sending a full tablet report...
I20260808 04:03:54.042829 6884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:54.030951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.3:33067 (local address 127.0.0.1:54142)
0808 04:03:54.032607 (+ 1656us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:54.032639 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0808 04:03:54.032839 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:03:54.033891 (+ 1052us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:03:54.033911 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:03:54.033954 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:03:54.034372 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:54.034386 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:54.036671 (+ 2285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:54.036677 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:54.039867 (+ 3190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:54.039881 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:54.041836 (+ 1955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:54.041842 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:54.041993 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:54.042030 (+ 37us) client_negotiation.cc:769] Sending connection context
0808 04:03:54.042196 (+ 166us) client_negotiation.cc:240] Negotiation successful
0808 04:03:54.042390 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1487,"mutex_wait_us":525,"thread_start_us":2066,"threads_started":1}
I20260808 04:03:54.042856 6881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:54.030958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49230 (local address 127.5.83.2:45547)
0808 04:03:54.031573 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:54.031582 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:54.033391 (+ 1809us) server_negotiation.cc:407] Connection header received
0808 04:03:54.033702 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:54.033711 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:54.033942 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:54.034139 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:03:54.035139 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:54.036807 (+ 1668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:54.039588 (+ 2781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:54.040976 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:54.041075 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:54.041255 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:54.042421 (+ 1166us) server_negotiation.cc:299] Negotiation successful
0808 04:03:54.042583 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":501,"threads_started":1}
I20260808 04:03:54.043284 6885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:54.031319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54142 (local address 127.5.83.3:33067)
0808 04:03:54.032191 (+ 872us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:54.032197 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:54.032741 (+ 544us) server_negotiation.cc:407] Connection header received
0808 04:03:54.033109 (+ 368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:54.033117 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:54.033350 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:54.033526 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:03:54.034608 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:54.036465 (+ 1857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:54.040039 (+ 3574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:54.041363 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:54.041492 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:54.041703 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:54.042439 (+ 736us) server_negotiation.cc:299] Negotiation successful
0808 04:03:54.042600 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":724,"thread_start_us":585,"threads_started":1}
I20260808 04:03:54.043511 6883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:54.030951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.2:45547 (local address 127.0.0.1:49230)
0808 04:03:54.033280 (+ 2329us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:54.033303 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 04:03:54.033497 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:03:54.034412 (+ 915us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:03:54.034420 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:03:54.034448 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:03:54.034934 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:54.034945 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:54.037017 (+ 2072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:54.037023 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:54.039398 (+ 2375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:54.039414 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:54.041438 (+ 2024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:54.041444 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:54.041643 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:54.042256 (+ 613us) client_negotiation.cc:769] Sending connection context
0808 04:03:54.043019 (+ 763us) client_negotiation.cc:240] Negotiation successful
0808 04:03:54.043141 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2216,"mutex_wait_us":220,"thread_start_us":1127,"threads_started":1}
I20260808 04:03:54.046857 6577 tablet_service.cc:1467] Processing CreateTablet for tablet 39c97a03304f495c8aeed8ec0588d3b5 (DEFAULT_TABLE table=test-table [id=99578852275e4f0a97467067f30671ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:03:54.047494 6886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:54.033406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:40423 (local address 127.0.0.1:53816)
0808 04:03:54.039970 (+ 6564us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:54.039991 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 04:03:54.040144 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:03:54.040579 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:03:54.040583 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:03:54.040606 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:03:54.040971 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:54.040984 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:54.043726 (+ 2742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:54.043732 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:54.045181 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:54.045194 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:54.046792 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:54.046798 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:54.046954 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:54.046988 (+ 34us) client_negotiation.cc:769] Sending connection context
0808 04:03:54.047070 (+ 82us) client_negotiation.cc:240] Negotiation successful
0808 04:03:54.047188 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6441,"thread_start_us":6304,"threads_started":1}
I20260808 04:03:54.047552 6887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:54.033938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53816 (local address 127.5.83.1:40423)
0808 04:03:54.039683 (+ 5745us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:54.039690 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:54.040108 (+ 418us) server_negotiation.cc:407] Connection header received
0808 04:03:54.040266 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:54.040273 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:54.040423 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:54.040572 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:03:54.041136 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:54.043535 (+ 2399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:54.045328 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:54.046599 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:54.046715 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:54.046937 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:54.047120 (+ 183us) server_negotiation.cc:299] Negotiation successful
0808 04:03:54.047269 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5592,"thread_start_us":5456,"threads_started":1}
I20260808 04:03:54.048568 6577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39c97a03304f495c8aeed8ec0588d3b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:03:54.049387 6770 tablet_service.cc:1467] Processing CreateTablet for tablet 39c97a03304f495c8aeed8ec0588d3b5 (DEFAULT_TABLE table=test-table [id=99578852275e4f0a97467067f30671ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:03:54.050737 6770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39c97a03304f495c8aeed8ec0588d3b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:03:54.054350 6393 tablet_service.cc:1467] Processing CreateTablet for tablet 39c97a03304f495c8aeed8ec0588d3b5 (DEFAULT_TABLE table=test-table [id=99578852275e4f0a97467067f30671ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:03:54.055399 6393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39c97a03304f495c8aeed8ec0588d3b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:03:54.063571 6888 tablet_bootstrap.cc:492] T 39c97a03304f495c8aeed8ec0588d3b5 P f21c4e80bd0d46919e76b7237ac9c714: Bootstrap starting.
I20260808 04:03:54.067423 6890 tablet_bootstrap.cc:492] T 39c97a03304f495c8aeed8ec0588d3b5 P 360e3415fb794cfba2f30ada9f50fa7a: Bootstrap starting.
I20260808 04:03:54.068521 6888 tablet_bootstrap.cc:654] T 39c97a03304f495c8aeed8ec0588d3b5 P f21c4e80bd0d46919e76b7237ac9c714: Neither blocks nor log segments found. Creating new log.
I20260808 04:03:54.068813 6889 tablet_bootstrap.cc:492] T 39c97a03304f495c8aeed8ec0588d3b5 P ac18d15cd5d143c78fb00dc74e5d8802: Bootstrap starting.
I20260808 04:03:54.069980 6888 log.cc:824] T 39c97a03304f495c8aeed8ec0588d3b5 P f21c4e80bd0d46919e76b7237ac9c714: Log is configured to *not* fsync() on all Append() calls
I20260808 04:03:54.070673 6890 tablet_bootstrap.cc:654] T 39c97a03304f495c8aeed8ec0588d3b5 P 360e3415fb794cfba2f30ada9f50fa7a: Neither blocks nor log segments found. Creating new log.
I20260808 04:03:54.071957 6890 log.cc:824] T 39c97a03304f495c8aeed8ec0588d3b5 P 360e3415fb794cfba2f30ada9f50fa7a: Log is configured to *not* fsync() on all Append() calls
I20260808 04:03:54.072288 6889 tablet_bootstrap.cc:654] T 39c97a03304f495c8aeed8ec0588d3b5 P ac18d15cd5d143c78fb00dc74e5d8802: Neither blocks nor log segments found. Creating new log.
I20260808 04:03:54.073509 6888 tablet_bootstrap.cc:492] T 39c97a03304f495c8aeed8ec0588d3b5 P f21c4e80bd0d46919e76b7237ac9c714: No bootstrap required, opened a new log
I20260808 04:03:54.073606 6889 log.cc:824] T 39c97a03304f495c8aeed8ec0588d3b5 P ac18d15cd5d143c78fb00dc74e5d8802: Log is configured to *not* fsync() on all Append() calls
I20260808 04:03:54.073783 6888 ts_tablet_manager.cc:1397] T 39c97a03304f495c8aeed8ec0588d3b5 P f21c4e80bd0d46919e76b7237ac9c714: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.007s
I20260808 04:03:54.077162 6889 tablet_bootstrap.cc:492] T 39c97a03304f495c8aeed8ec0588d3b5 P ac18d15cd5d143c78fb00dc74e5d8802: No bootstrap required, opened a new log
I20260808 04:03:54.077425 6889 ts_tablet_manager.cc:1397] T 39c97a03304f495c8aeed8ec0588d3b5 P ac18d15cd5d143c78fb00dc74e5d8802: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260808 04:03:54.077780 6890 tablet_bootstrap.cc:492] T 39c97a03304f495c8aeed8ec0588d3b5 P 360e3415fb794cfba2f30ada9f50fa7a: No bootstrap required, opened a new log
I20260808 04:03:54.078037 6890 ts_tablet_manager.cc:1397] T 39c97a03304f495c8aeed8ec0588d3b5 P 360e3415fb794cfba2f30ada9f50fa7a: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20260808 04:03:54.084549 6888 raft_consensus.cc:348] T 39c97a03304f495c8aeed8ec0588d3b5 P f21c4e80bd0d46919e76b7237ac9c714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c4e80bd0d46919e76b7237ac9c714" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 45547 } } peers { permanent_uuid: "ac18d15cd5d143c78fb00dc74e5d8802" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 33067 } } peers { permanent_uuid: "360e3415fb794cfba2f30ada9f50fa7a" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 40423 } }
I20260808 04:03:54.085037 6888 raft_consensus.cc:374] T 39c97a03304f495c8aeed8ec0588d3b5 P f21c4e80bd0d46919e76b7237ac9c714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:03:54.085146 6888 raft_consensus.cc:729] T 39c97a03304f495c8aeed8ec0588d3b5 P f21c4e80bd0d46919e76b7237ac9c714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c4e80bd0d46919e76b7237ac9c714, State: Initialized, Role: FOLLOWER
I20260808 04:03:54.084975 6890 raft_consensus.cc:348] T 39c97a03304f495c8aeed8ec0588d3b5 P 360e3415fb794cfba2f30ada9f50fa7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c4e80bd0d46919e76b7237ac9c714" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 45547 } } peers { permanent_uuid: "ac18d15cd5d143c78fb00dc74e5d8802" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 33067 } } peers { permanent_uuid: "360e3415fb794cfba2f30ada9f50fa7a" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 40423 } }
I20260808 04:03:54.085371 6890 raft_consensus.cc:374] T 39c97a03304f495c8aeed8ec0588d3b5 P 360e3415fb794cfba2f30ada9f50fa7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:03:54.085462 6890 raft_consensus.cc:729] T 39c97a03304f495c8aeed8ec0588d3b5 P 360e3415fb794cfba2f30ada9f50fa7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 360e3415fb794cfba2f30ada9f50fa7a, State: Initialized, Role: FOLLOWER
I20260808 04:03:54.085916 6888 consensus_queue.cc:260] T 39c97a03304f495c8aeed8ec0588d3b5 P f21c4e80bd0d46919e76b7237ac9c714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c4e80bd0d46919e76b7237ac9c714" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 45547 } } peers { permanent_uuid: "ac18d15cd5d143c78fb00dc74e5d8802" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 33067 } } peers { permanent_uuid: "360e3415fb794cfba2f30ada9f50fa7a" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 40423 } }
I20260808 04:03:54.085974 6890 consensus_queue.cc:260] T 39c97a03304f495c8aeed8ec0588d3b5 P 360e3415fb794cfba2f30ada9f50fa7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c4e80bd0d46919e76b7237ac9c714" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 45547 } } peers { permanent_uuid: "ac18d15cd5d143c78fb00dc74e5d8802" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 33067 } } peers { permanent_uuid: "360e3415fb794cfba2f30ada9f50fa7a" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 40423 } }
I20260808 04:03:54.087683 6667 heartbeater.cc:503] Master 127.5.83.62:45043 was elected leader, sending a full tablet report...
I20260808 04:03:54.087550 6889 raft_consensus.cc:348] T 39c97a03304f495c8aeed8ec0588d3b5 P ac18d15cd5d143c78fb00dc74e5d8802 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c4e80bd0d46919e76b7237ac9c714" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 45547 } } peers { permanent_uuid: "ac18d15cd5d143c78fb00dc74e5d8802" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 33067 } } peers { permanent_uuid: "360e3415fb794cfba2f30ada9f50fa7a" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 40423 } }
I20260808 04:03:54.088013 6889 raft_consensus.cc:374] T 39c97a03304f495c8aeed8ec0588d3b5 P ac18d15cd5d143c78fb00dc74e5d8802 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:03:54.088114 6889 raft_consensus.cc:729] T 39c97a03304f495c8aeed8ec0588d3b5 P ac18d15cd5d143c78fb00dc74e5d8802 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac18d15cd5d143c78fb00dc74e5d8802, State: Initialized, Role: FOLLOWER
I20260808 04:03:54.088359 6888 ts_tablet_manager.cc:1428] T 39c97a03304f495c8aeed8ec0588d3b5 P f21c4e80bd0d46919e76b7237ac9c714: Time spent starting tablet: real 0.014s user 0.009s sys 0.006s
I20260808 04:03:54.088773 6889 consensus_queue.cc:260] T 39c97a03304f495c8aeed8ec0588d3b5 P ac18d15cd5d143c78fb00dc74e5d8802 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c4e80bd0d46919e76b7237ac9c714" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 45547 } } peers { permanent_uuid: "ac18d15cd5d143c78fb00dc74e5d8802" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 33067 } } peers { permanent_uuid: "360e3415fb794cfba2f30ada9f50fa7a" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 40423 } }
I20260808 04:03:54.089349 6890 ts_tablet_manager.cc:1428] T 39c97a03304f495c8aeed8ec0588d3b5 P 360e3415fb794cfba2f30ada9f50fa7a: Time spent starting tablet: real 0.011s user 0.009s sys 0.002s
I20260808 04:03:54.090219 6864 heartbeater.cc:503] Master 127.5.83.62:45043 was elected leader, sending a full tablet report...
I20260808 04:03:54.092337 6889 ts_tablet_manager.cc:1428] T 39c97a03304f495c8aeed8ec0588d3b5 P ac18d15cd5d143c78fb00dc74e5d8802: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
I20260808 04:03:54.195350 6898 raft_consensus.cc:482] T 39c97a03304f495c8aeed8ec0588d3b5 P f21c4e80bd0d46919e76b7237ac9c714 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:03:54.195691 6898 raft_consensus.cc:504] T 39c97a03304f495c8aeed8ec0588d3b5 P f21c4e80bd0d46919e76b7237ac9c714 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c4e80bd0d46919e76b7237ac9c714" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 45547 } } peers { permanent_uuid: "ac18d15cd5d143c78fb00dc74e5d8802" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 33067 } } peers { permanent_uuid: "360e3415fb794cfba2f30ada9f50fa7a" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 40423 } }
I20260808 04:03:54.197813 6898 leader_election.cc:288] T 39c97a03304f495c8aeed8ec0588d3b5 P f21c4e80bd0d46919e76b7237ac9c714 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac18d15cd5d143c78fb00dc74e5d8802 (127.5.83.3:33067), 360e3415fb794cfba2f30ada9f50fa7a (127.5.83.1:40423)
I20260808 04:03:54.204558 6885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:54.198120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.2:40325 (local address 127.5.83.3:33067)
0808 04:03:54.198339 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:54.198347 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:54.198753 (+ 406us) server_negotiation.cc:407] Connection header received
0808 04:03:54.198977 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:54.198982 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:54.199061 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:54.199147 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:03:54.199785 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:54.200766 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:54.202676 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:54.203554 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:54.203645 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:54.203748 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:54.204145 (+ 397us) server_negotiation.cc:299] Negotiation successful
0808 04:03:54.204267 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 04:03:54.204576 6900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:54.198031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.3:33067 (local address 127.5.83.2:40325)
0808 04:03:54.198646 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:54.198663 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:03:54.198817 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:03:54.199182 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:03:54.199187 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:03:54.199217 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:03:54.199624 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:54.199639 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:54.200891 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:54.200896 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:54.202524 (+ 1628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:54.202536 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:54.203795 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:54.203802 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:54.203969 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:54.204006 (+ 37us) client_negotiation.cc:769] Sending connection context
0808 04:03:54.204110 (+ 104us) client_negotiation.cc:240] Negotiation successful
0808 04:03:54.204242 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"mutex_wait_us":4,"thread_start_us":321,"threads_started":1}
I20260808 04:03:54.205034 6901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:54.198710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:40423 (local address 127.5.83.2:33697)
0808 04:03:54.199430 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:54.199444 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:03:54.199557 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:03:54.199897 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:03:54.199902 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:03:54.199927 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:03:54.200294 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:54.200305 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:54.201435 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:54.201438 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:54.202933 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:54.202947 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:54.204247 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:54.204253 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:54.204460 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:54.204491 (+ 31us) client_negotiation.cc:769] Sending connection context
0808 04:03:54.204597 (+ 106us) client_negotiation.cc:240] Negotiation successful
0808 04:03:54.204720 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":386,"threads_started":1}
I20260808 04:03:54.205039 6887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:54.198877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.2:33697 (local address 127.5.83.1:40423)
0808 04:03:54.199088 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:54.199095 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:54.199535 (+ 440us) server_negotiation.cc:407] Connection header received
0808 04:03:54.199675 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:54.199680 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:54.199781 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:54.199852 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:03:54.200447 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:54.201287 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:54.203086 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:54.204028 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:54.204142 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:54.204235 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:54.204657 (+ 422us) server_negotiation.cc:299] Negotiation successful
0808 04:03:54.204786 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 04:03:54.205691 6800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39c97a03304f495c8aeed8ec0588d3b5" candidate_uuid: "f21c4e80bd0d46919e76b7237ac9c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac18d15cd5d143c78fb00dc74e5d8802" is_pre_election: true
I20260808 04:03:54.206067 6422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39c97a03304f495c8aeed8ec0588d3b5" candidate_uuid: "f21c4e80bd0d46919e76b7237ac9c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "360e3415fb794cfba2f30ada9f50fa7a" is_pre_election: true
I20260808 04:03:54.206430 6800 raft_consensus.cc:2454] T 39c97a03304f495c8aeed8ec0588d3b5 P ac18d15cd5d143c78fb00dc74e5d8802 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f21c4e80bd0d46919e76b7237ac9c714 in term 0.
I20260808 04:03:54.206528 6422 raft_consensus.cc:2454] T 39c97a03304f495c8aeed8ec0588d3b5 P 360e3415fb794cfba2f30ada9f50fa7a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f21c4e80bd0d46919e76b7237ac9c714 in term 0.
I20260808 04:03:54.207196 6511 leader_election.cc:302] T 39c97a03304f495c8aeed8ec0588d3b5 P f21c4e80bd0d46919e76b7237ac9c714 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac18d15cd5d143c78fb00dc74e5d8802, f21c4e80bd0d46919e76b7237ac9c714; no voters:
I20260808 04:03:54.207535 6898 raft_consensus.cc:2781] T 39c97a03304f495c8aeed8ec0588d3b5 P f21c4e80bd0d46919e76b7237ac9c714 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:03:54.207684 6898 raft_consensus.cc:482] T 39c97a03304f495c8aeed8ec0588d3b5 P f21c4e80bd0d46919e76b7237ac9c714 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:03:54.207827 6898 raft_consensus.cc:3037] T 39c97a03304f495c8aeed8ec0588d3b5 P f21c4e80bd0d46919e76b7237ac9c714 [term 0 FOLLOWER]: Advancing to term 1
W20260808 04:03:54.208669 6865 tablet.cc:2369] T 39c97a03304f495c8aeed8ec0588d3b5 P ac18d15cd5d143c78fb00dc74e5d8802: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:03:54.210420 6898 raft_consensus.cc:504] T 39c97a03304f495c8aeed8ec0588d3b5 P f21c4e80bd0d46919e76b7237ac9c714 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c4e80bd0d46919e76b7237ac9c714" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 45547 } } peers { permanent_uuid: "ac18d15cd5d143c78fb00dc74e5d8802" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 33067 } } peers { permanent_uuid: "360e3415fb794cfba2f30ada9f50fa7a" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 40423 } }
I20260808 04:03:54.210955 6898 leader_election.cc:288] T 39c97a03304f495c8aeed8ec0588d3b5 P f21c4e80bd0d46919e76b7237ac9c714 [CANDIDATE]: Term 1 election: Requested vote from peers ac18d15cd5d143c78fb00dc74e5d8802 (127.5.83.3:33067), 360e3415fb794cfba2f30ada9f50fa7a (127.5.83.1:40423)
I20260808 04:03:54.211374 6800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39c97a03304f495c8aeed8ec0588d3b5" candidate_uuid: "f21c4e80bd0d46919e76b7237ac9c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac18d15cd5d143c78fb00dc74e5d8802"
I20260808 04:03:54.211450 6422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39c97a03304f495c8aeed8ec0588d3b5" candidate_uuid: "f21c4e80bd0d46919e76b7237ac9c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "360e3415fb794cfba2f30ada9f50fa7a"
I20260808 04:03:54.211556 6800 raft_consensus.cc:3037] T 39c97a03304f495c8aeed8ec0588d3b5 P ac18d15cd5d143c78fb00dc74e5d8802 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:03:54.211642 6422 raft_consensus.cc:3037] T 39c97a03304f495c8aeed8ec0588d3b5 P 360e3415fb794cfba2f30ada9f50fa7a [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:03:54.213690 6800 raft_consensus.cc:2454] T 39c97a03304f495c8aeed8ec0588d3b5 P ac18d15cd5d143c78fb00dc74e5d8802 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f21c4e80bd0d46919e76b7237ac9c714 in term 1.
I20260808 04:03:54.213958 6422 raft_consensus.cc:2454] T 39c97a03304f495c8aeed8ec0588d3b5 P 360e3415fb794cfba2f30ada9f50fa7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f21c4e80bd0d46919e76b7237ac9c714 in term 1.
I20260808 04:03:54.214084 6511 leader_election.cc:302] T 39c97a03304f495c8aeed8ec0588d3b5 P f21c4e80bd0d46919e76b7237ac9c714 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac18d15cd5d143c78fb00dc74e5d8802, f21c4e80bd0d46919e76b7237ac9c714; no voters:
I20260808 04:03:54.214445 6898 raft_consensus.cc:2781] T 39c97a03304f495c8aeed8ec0588d3b5 P f21c4e80bd0d46919e76b7237ac9c714 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:03:54.214854 6898 raft_consensus.cc:686] T 39c97a03304f495c8aeed8ec0588d3b5 P f21c4e80bd0d46919e76b7237ac9c714 [term 1 LEADER]: Becoming Leader. State: Replica: f21c4e80bd0d46919e76b7237ac9c714, State: Running, Role: LEADER
I20260808 04:03:54.215603 6898 consensus_queue.cc:237] T 39c97a03304f495c8aeed8ec0588d3b5 P f21c4e80bd0d46919e76b7237ac9c714 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c4e80bd0d46919e76b7237ac9c714" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 45547 } } peers { permanent_uuid: "ac18d15cd5d143c78fb00dc74e5d8802" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 33067 } } peers { permanent_uuid: "360e3415fb794cfba2f30ada9f50fa7a" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 40423 } }
I20260808 04:03:54.220726 6220 catalog_manager.cc:5495] T 39c97a03304f495c8aeed8ec0588d3b5 P f21c4e80bd0d46919e76b7237ac9c714 reported cstate change: term changed from 0 to 1, leader changed from <none> to f21c4e80bd0d46919e76b7237ac9c714 (127.5.83.2). New cstate: current_term: 1 leader_uuid: "f21c4e80bd0d46919e76b7237ac9c714" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c4e80bd0d46919e76b7237ac9c714" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 45547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac18d15cd5d143c78fb00dc74e5d8802" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 33067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "360e3415fb794cfba2f30ada9f50fa7a" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 40423 } health_report { overall_health: UNKNOWN } } }
W20260808 04:03:54.251521 6669 tablet.cc:2369] T 39c97a03304f495c8aeed8ec0588d3b5 P f21c4e80bd0d46919e76b7237ac9c714: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 04:03:54.255491 6486 tablet.cc:2369] T 39c97a03304f495c8aeed8ec0588d3b5 P 360e3415fb794cfba2f30ada9f50fa7a: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 08 04:03:54 dist-test-slave-1l3l krb5kdc[6164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.1@KRBTEST.COM
I20260808 04:03:54.279883 6887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:54.265569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53824 (local address 127.5.83.1:40423)
0808 04:03:54.265747 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:54.265752 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:54.266242 (+ 490us) server_negotiation.cc:407] Connection header received
0808 04:03:54.266436 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:54.266440 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:54.266522 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:54.266608 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:54.268123 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:54.268905 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:54.270120 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:54.270594 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:54.270707 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:54.274587 (+ 3880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:54.274600 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:54.274614 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:03:54.274650 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:54.277443 (+ 2793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:54.278041 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:54.278049 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:54.278051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:54.278178 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:54.278562 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:54.278566 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:54.278568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:54.278920 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:54.279133 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:54.279370 (+ 237us) server_negotiation.cc:299] Negotiation successful
0808 04:03:54.279535 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
Aug 08 04:03:54 dist-test-slave-1l3l krb5kdc[6164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161834, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 04:03:54 dist-test-slave-1l3l krb5kdc[6164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161834, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.83.1@KRBTEST.COM
I20260808 04:03:54.325664 6887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:54.312908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53838 (local address 127.5.83.1:40423)
0808 04:03:54.313120 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:54.313128 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:54.313490 (+ 362us) server_negotiation.cc:407] Connection header received
0808 04:03:54.313662 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:54.313666 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:54.313729 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:54.313814 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:54.315124 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:54.315936 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:54.317182 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:54.317613 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:54.317725 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:54.321261 (+ 3536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:54.321265 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:54.321269 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:03:54.321301 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:54.323633 (+ 2332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:54.324202 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:54.324206 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:54.324208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:54.324336 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:54.324636 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:54.324639 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:54.324641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:54.324872 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:54.325071 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:54.325241 (+ 170us) server_negotiation.cc:299] Negotiation successful
0808 04:03:54.325421 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
Aug 08 04:03:54 dist-test-slave-1l3l krb5kdc[6164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161834, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 04:03:54 dist-test-slave-1l3l krb5kdc[6164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161834, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.1@KRBTEST.COM
I20260808 04:03:54.371869 6887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:54.358504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53848 (local address 127.5.83.1:40423)
0808 04:03:54.358744 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:54.358749 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:54.358876 (+ 127us) server_negotiation.cc:407] Connection header received
0808 04:03:54.359026 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:54.359030 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:54.359094 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:54.359177 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:54.360565 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:54.361271 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:54.362520 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:54.362941 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:54.363036 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:54.366950 (+ 3914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:54.366956 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:54.366959 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:03:54.366992 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:54.369287 (+ 2295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:54.369932 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:54.369936 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:54.369938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:54.370057 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:54.370406 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:54.370411 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:54.370414 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:54.370739 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:54.370984 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:54.371252 (+ 268us) server_negotiation.cc:299] Negotiation successful
0808 04:03:54.371501 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260808 04:03:54.374179 6367 tablet_service.cc:2908] Rejecting scan request for tablet 39c97a03304f495c8aeed8ec0588d3b5: Uninitialized: clean time has not yet been initialized
I20260808 04:03:54.378211 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 6309
I20260808 04:03:54.389442 6887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:54.379034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53858 (local address 127.5.83.1:40423)
0808 04:03:54.379180 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:54.379184 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:54.379476 (+ 292us) server_negotiation.cc:407] Connection header received
0808 04:03:54.379576 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:54.379580 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:54.379647 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:54.379763 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:54.380836 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:54.381507 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:54.382632 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:54.383047 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:54.383148 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:54.385116 (+ 1968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:54.385121 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:54.385124 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:03:54.385164 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:54.387413 (+ 2249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:54.388042 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:54.388047 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:54.388049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:54.388166 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:54.388434 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:54.388437 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:54.388439 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:54.388664 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:54.388890 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:54.389057 (+ 167us) server_negotiation.cc:299] Negotiation successful
0808 04:03:54.389212 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260808 04:03:54.390017 6478 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:03:54.476205 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 6309
I20260808 04:03:54.497032 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 6491
Aug 08 04:03:54 dist-test-slave-1l3l krb5kdc[6164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161834, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.2@KRBTEST.COM
I20260808 04:03:54.511809 6881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:54.497866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49238 (local address 127.5.83.2:45547)
0808 04:03:54.498081 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:54.498086 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:54.498106 (+ 20us) server_negotiation.cc:407] Connection header received
0808 04:03:54.498181 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:54.498186 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:54.498267 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:54.498353 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:54.499555 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:54.500339 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:54.501595 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:54.502015 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:54.502127 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:54.506293 (+ 4166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:54.506303 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:54.506316 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:03:54.506348 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:54.509302 (+ 2954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:54.510063 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:54.510070 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:54.510072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:54.510219 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:54.510576 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:54.510579 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:54.510581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:54.510899 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:54.511115 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:54.511309 (+ 194us) server_negotiation.cc:299] Negotiation successful
0808 04:03:54.511485 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260808 04:03:54.512486 6659 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:03:54.599588 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 6491
I20260808 04:03:54.620692 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 6673
Aug 08 04:03:54 dist-test-slave-1l3l krb5kdc[6164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161834, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.3@KRBTEST.COM
I20260808 04:03:54.634979 6885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:54.621479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54146 (local address 127.5.83.3:33067)
0808 04:03:54.621660 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:54.621665 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:54.621687 (+ 22us) server_negotiation.cc:407] Connection header received
0808 04:03:54.621866 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:54.621871 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:54.621947 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:54.622030 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:54.623294 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:54.624091 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:54.625351 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:54.625744 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:54.625854 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:54.629816 (+ 3962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:54.629832 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:54.629848 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:03:54.629899 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:54.632396 (+ 2497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:54.633009 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:54.633018 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:54.633020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:54.633151 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:54.633511 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:54.633515 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:54.633517 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:54.633855 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:54.634136 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:54.634410 (+ 274us) server_negotiation.cc:299] Negotiation successful
0808 04:03:54.634639 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 04:03:54.635640 6859 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:03:54.720309 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 6673
I20260808 04:03:54.739373 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 6180
I20260808 04:03:54.740569 6269 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:03:54.847859 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 6180
2026-08-08T04:03:54Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4175 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:03:54 dist-test-slave-1l3l krb5kdc[6923](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:03:54 dist-test-slave-1l3l krb5kdc[6923](info): set up 2 sockets
Aug 08 04:03:54 dist-test-slave-1l3l krb5kdc[6923](info): commencing operation
krb5kdc: starting...
W20260808 04:03:56.002758 5452 mini_kdc.cc:121] Time spent starting KDC: real 1.111s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/krb5kdc/test-user.keytab.
Aug 08 04:03:56 dist-test-slave-1l3l krb5kdc[6923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:03:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:03:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.83.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab.
Entry for principal kudu/127.5.83.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab.
I20260808 04:03:56.192847 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:45699
--webserver_interface=127.5.83.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab
--principal=kudu/127.5.83.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:37307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.62:45699
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:03:56.506925 6942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:03:56.507182 6942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:03:56.507309 6942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:03:56.514252 6942 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:03:56.514355 6942 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260808 04:03:56.514420 6942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:03:56.514501 6942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:03:56.514566 6942 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:03:56.514649 6942 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:03:56.524032 6942 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:37307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.62:45699
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab
--principal=kudu/127.5.83.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:45699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:03:56.525154 6942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:03:56.526479 6942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:03:56.537282 6950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:03:56.537367 6949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:03:56.537853 6954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:03:56.538287 6942 server_base.cc:1034] running on GCE node
I20260808 04:03:56.539028 6942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:03:56.540484 6942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:03:56.541697 6942 hybrid_clock.cc:648] HybridClock initialized: now 1786161836541642 us; error 32 us; skew 500 ppm
I20260808 04:03:56.541994 6942 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:03:56 dist-test-slave-1l3l krb5kdc[6923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161836, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:03:56.544952 6942 init.cc:377] Logged in from keytab as kudu/127.5.83.62@KRBTEST.COM (short username kudu)
I20260808 04:03:56.546223 6942 webserver.cc:460] Webserver started at http://127.5.83.62:40881/ using document root <none> and password file <none>
I20260808 04:03:56.546875 6942 fs_manager.cc:359] Metadata directory not provided
I20260808 04:03:56.546976 6942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:03:56.547277 6942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:03:56.550113 6942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/minicluster-data/master-0/data/instance:
uuid: "0f18f42ae17b4626a802d0e17926c51a"
format_stamp: "Formatted at 2026-08-08 04:03:56 on dist-test-slave-1l3l"
server_key: "c6dcbedf78bc16e8f4b6fa7a0ffc9558"
server_key_iv: "d65acf3bb31893531c4596a38d59cb6b"
server_key_version: "encryptionkey@0"
I20260808 04:03:56.550860 6942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/minicluster-data/master-0/wal/instance:
uuid: "0f18f42ae17b4626a802d0e17926c51a"
format_stamp: "Formatted at 2026-08-08 04:03:56 on dist-test-slave-1l3l"
server_key: "c6dcbedf78bc16e8f4b6fa7a0ffc9558"
server_key_iv: "d65acf3bb31893531c4596a38d59cb6b"
server_key_version: "encryptionkey@0"
I20260808 04:03:56.555517 6942 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 04:03:56.558693 6962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:03:56.559912 6942 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:03:56.560055 6942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/minicluster-data/master-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/minicluster-data/master-0/wal
uuid: "0f18f42ae17b4626a802d0e17926c51a"
format_stamp: "Formatted at 2026-08-08 04:03:56 on dist-test-slave-1l3l"
server_key: "c6dcbedf78bc16e8f4b6fa7a0ffc9558"
server_key_iv: "d65acf3bb31893531c4596a38d59cb6b"
server_key_version: "encryptionkey@0"
I20260808 04:03:56.560232 6942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:03:56.570320 6942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:03:56.573827 6942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:03:56.574054 6942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:03:56.594343 6942 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.62:45699
I20260808 04:03:56.594359 7048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.62:45699 every 8 connection(s)
I20260808 04:03:56.595643 6942 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
I20260808 04:03:56.599141 7049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:03:56.600656 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 6942
I20260808 04:03:56.600867 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/minicluster-data/master-0/wal/instance
I20260808 04:03:56.601501 5452 external_mini_cluster.cc:1428] Setting key ecf694f552963cc2de9cd05025d6bf72
I20260808 04:03:56.610222 7049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f18f42ae17b4626a802d0e17926c51a: Bootstrap starting.
I20260808 04:03:56.613895 7049 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f18f42ae17b4626a802d0e17926c51a: Neither blocks nor log segments found. Creating new log.
I20260808 04:03:56.615108 7049 log.cc:824] T 00000000000000000000000000000000 P 0f18f42ae17b4626a802d0e17926c51a: Log is configured to *not* fsync() on all Append() calls
Aug 08 04:03:56 dist-test-slave-1l3l krb5kdc[6923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:03:56.617928 7049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f18f42ae17b4626a802d0e17926c51a: No bootstrap required, opened a new log
I20260808 04:03:56.623306 7053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:56.603334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36534 (local address 127.5.83.62:45699)
0808 04:03:56.604099 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:56.604113 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:56.604155 (+ 42us) server_negotiation.cc:407] Connection header received
0808 04:03:56.605776 (+ 1621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:56.605797 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:56.606132 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:56.606442 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:56.608142 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:56.609453 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:56.611109 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:56.611826 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:56.611958 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:56.616645 (+ 4687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:56.616652 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:56.616665 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:03:56.616707 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:56.619581 (+ 2874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:56.620235 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:56.620241 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:56.620246 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:56.620433 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:56.620785 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:56.620792 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:56.620794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:56.621290 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:56.621603 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:56.621978 (+ 375us) server_negotiation.cc:299] Negotiation successful
0808 04:03:56.622372 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":360,"threads_started":1}
I20260808 04:03:56.625857 7049 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f18f42ae17b4626a802d0e17926c51a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f18f42ae17b4626a802d0e17926c51a" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 45699 } }
I20260808 04:03:56.626256 7049 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f18f42ae17b4626a802d0e17926c51a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:03:56.626350 7049 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f18f42ae17b4626a802d0e17926c51a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f18f42ae17b4626a802d0e17926c51a, State: Initialized, Role: FOLLOWER
I20260808 04:03:56.626838 7049 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f18f42ae17b4626a802d0e17926c51a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f18f42ae17b4626a802d0e17926c51a" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 45699 } }
I20260808 04:03:56.627044 7049 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f18f42ae17b4626a802d0e17926c51a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:03:56.627144 7049 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f18f42ae17b4626a802d0e17926c51a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:03:56.627257 7049 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f18f42ae17b4626a802d0e17926c51a [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:03:56.629628 7049 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f18f42ae17b4626a802d0e17926c51a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f18f42ae17b4626a802d0e17926c51a" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 45699 } }
I20260808 04:03:56.630048 7049 leader_election.cc:302] T 00000000000000000000000000000000 P 0f18f42ae17b4626a802d0e17926c51a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f18f42ae17b4626a802d0e17926c51a; no voters:
I20260808 04:03:56.630530 7049 leader_election.cc:288] T 00000000000000000000000000000000 P 0f18f42ae17b4626a802d0e17926c51a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:03:56.630640 7055 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f18f42ae17b4626a802d0e17926c51a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:03:56.631318 7055 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f18f42ae17b4626a802d0e17926c51a [term 1 LEADER]: Becoming Leader. State: Replica: 0f18f42ae17b4626a802d0e17926c51a, State: Running, Role: LEADER
I20260808 04:03:56.631961 7055 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f18f42ae17b4626a802d0e17926c51a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f18f42ae17b4626a802d0e17926c51a" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 45699 } }
I20260808 04:03:56.632577 7049 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f18f42ae17b4626a802d0e17926c51a [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:03:56.634533 7057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f18f42ae17b4626a802d0e17926c51a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f18f42ae17b4626a802d0e17926c51a. Latest consensus state: current_term: 1 leader_uuid: "0f18f42ae17b4626a802d0e17926c51a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f18f42ae17b4626a802d0e17926c51a" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 45699 } } }
I20260808 04:03:56.634760 7056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f18f42ae17b4626a802d0e17926c51a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0f18f42ae17b4626a802d0e17926c51a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f18f42ae17b4626a802d0e17926c51a" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 45699 } } }
I20260808 04:03:56.634932 7057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f18f42ae17b4626a802d0e17926c51a [sys.catalog]: This master's current role is: LEADER
I20260808 04:03:56.634995 7056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f18f42ae17b4626a802d0e17926c51a [sys.catalog]: This master's current role is: LEADER
I20260808 04:03:56.636610 7066 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:03:56.641129 7066 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:03:56.649260 7066 catalog_manager.cc:1347] Generated new cluster ID: 32153e5d659448e5b57593bc2fd6b682
I20260808 04:03:56.649362 7066 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:03:56.660503 7066 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:03:56.661684 7066 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:03:56.672320 7066 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f18f42ae17b4626a802d0e17926c51a: Generated new TSK 0
I20260808 04:03:56.673048 7066 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.83.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.1.keytab.
Entry for principal kudu/127.5.83.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.1.keytab.
I20260808 04:03:56.728878 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.1:0
--local_ip_for_outbound_sockets=127.5.83.1
--webserver_interface=127.5.83.1
--webserver_port=0
--tserver_master_addrs=127.5.83.62:45699
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.1.keytab
--principal=kudu/127.5.83.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:37307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:03:57.066167 7084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:03:57.066442 7084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:03:57.066538 7084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:03:57.073537 7084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:03:57.073633 7084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:03:57.073717 7084 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 04:03:57.073794 7084 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.1
I20260808 04:03:57.083124 7084 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:37307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.1.keytab
--principal=kudu/127.5.83.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.83.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.83.62:45699
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.1
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:03:57.084380 7084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:03:57.085676 7084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:03:57.097731 7095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:03:57.097832 7093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:03:57.097922 7091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:03:57.098717 7084 server_base.cc:1034] running on GCE node
I20260808 04:03:57.099256 7084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:03:57.100106 7084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:03:57.101326 7084 hybrid_clock.cc:648] HybridClock initialized: now 1786161837101262 us; error 33 us; skew 500 ppm
I20260808 04:03:57.101598 7084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:03:57 dist-test-slave-1l3l krb5kdc[6923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161837, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:03:57.104615 7084 init.cc:377] Logged in from keytab as kudu/127.5.83.1@KRBTEST.COM (short username kudu)
I20260808 04:03:57.105856 7084 webserver.cc:460] Webserver started at http://127.5.83.1:38225/ using document root <none> and password file <none>
I20260808 04:03:57.106459 7084 fs_manager.cc:359] Metadata directory not provided
I20260808 04:03:57.106544 7084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:03:57.106805 7084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:03:57.109511 7084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/minicluster-data/ts-0/data/instance:
uuid: "5fcb22667c684fa495415f7f391266c1"
format_stamp: "Formatted at 2026-08-08 04:03:57 on dist-test-slave-1l3l"
server_key: "e009de2d9595d9a8de24697aa127657e"
server_key_iv: "707f43da9333adba7c56c9792f513309"
server_key_version: "encryptionkey@0"
I20260808 04:03:57.110227 7084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/minicluster-data/ts-0/wal/instance:
uuid: "5fcb22667c684fa495415f7f391266c1"
format_stamp: "Formatted at 2026-08-08 04:03:57 on dist-test-slave-1l3l"
server_key: "e009de2d9595d9a8de24697aa127657e"
server_key_iv: "707f43da9333adba7c56c9792f513309"
server_key_version: "encryptionkey@0"
I20260808 04:03:57.114949 7084 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260808 04:03:57.118079 7106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:03:57.119238 7084 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260808 04:03:57.119365 7084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/minicluster-data/ts-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/minicluster-data/ts-0/wal
uuid: "5fcb22667c684fa495415f7f391266c1"
format_stamp: "Formatted at 2026-08-08 04:03:57 on dist-test-slave-1l3l"
server_key: "e009de2d9595d9a8de24697aa127657e"
server_key_iv: "707f43da9333adba7c56c9792f513309"
server_key_version: "encryptionkey@0"
I20260808 04:03:57.119544 7084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:03:57.139154 7084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:03:57.142982 7084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:03:57.143195 7084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:03:57.145593 7084 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:03:57.145670 7084 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:03:57.145794 7084 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:03:57.145850 7084 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 04:03:57 dist-test-slave-1l3l krb5kdc[6923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161837, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.1@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:03:57.166738 7120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:57.146309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.1:51845 (local address 127.5.83.62:45699)
0808 04:03:57.148398 (+ 2089us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:57.148404 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:57.148426 (+ 22us) server_negotiation.cc:407] Connection header received
0808 04:03:57.149928 (+ 1502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:57.149934 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:57.150013 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:57.150107 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:57.152082 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:57.153020 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:57.155528 (+ 2508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:57.155972 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:57.156082 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:57.160209 (+ 4127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:57.160214 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:57.160217 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:03:57.160250 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:57.163193 (+ 2943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:57.164090 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:57.164094 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:57.164096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:57.164256 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:57.164787 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:57.164792 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:57.164795 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:57.165162 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:57.165296 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:57.166171 (+ 875us) server_negotiation.cc:299] Negotiation successful
0808 04:03:57.166386 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1973,"thread_start_us":1877,"threads_started":1}
I20260808 04:03:57.168077 7121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:57.146804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:45699 (local address 127.5.83.1:51845)
0808 04:03:57.147662 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:57.147732 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0808 04:03:57.149658 (+ 1926us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:03:57.150372 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:03:57.150395 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:03:57.151096 (+ 701us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:03:57.151909 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:57.151932 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:57.153197 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:57.153201 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:57.155345 (+ 2144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:57.155361 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:57.156231 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:57.156244 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:57.156437 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:57.157105 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:03:57.157126 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:03:57.160009 (+ 2883us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:03:57.163331 (+ 3322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:03:57.163340 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:03:57.163354 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:03:57.163936 (+ 582us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:03:57.164384 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:03:57.164389 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:03:57.164393 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:03:57.164665 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:03:57.165289 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:03:57.165300 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:03:57.165714 (+ 414us) client_negotiation.cc:769] Sending connection context
0808 04:03:57.166070 (+ 356us) client_negotiation.cc:240] Negotiation successful
0808 04:03:57.166589 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":341,"threads_started":1}
I20260808 04:03:57.186408 7084 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.1:42131
I20260808 04:03:57.186472 7272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.1:42131 every 8 connection(s)
I20260808 04:03:57.187934 7084 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
I20260808 04:03:57.189623 7120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:57.175360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.1:49571 (local address 127.5.83.62:45699)
0808 04:03:57.175527 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:57.175532 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:57.175765 (+ 233us) server_negotiation.cc:407] Connection header received
0808 04:03:57.175926 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:57.175931 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:57.176011 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:57.176119 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:57.177424 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:57.178468 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:57.180689 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:57.181148 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:57.181245 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:57.183692 (+ 2447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:57.183699 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:57.183747 (+ 48us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:03:57.183788 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:57.186785 (+ 2997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:57.187564 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:57.187568 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:57.187570 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:57.187694 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:57.188137 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:57.188141 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:57.188143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:57.188393 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:57.188566 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:57.189075 (+ 509us) server_negotiation.cc:299] Negotiation successful
0808 04:03:57.189283 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 04:03:57.189735 7232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:57.175047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:45699 (local address 127.5.83.1:49571)
0808 04:03:57.175629 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:57.175647 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 04:03:57.175797 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:03:57.176162 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:03:57.176167 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:03:57.176808 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:03:57.177267 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:57.177282 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:57.178653 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:57.178660 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:57.180533 (+ 1873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:57.180551 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:57.181339 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:57.181347 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:57.181552 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:57.182220 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:03:57.182246 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:03:57.183499 (+ 1253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:03:57.186899 (+ 3400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:03:57.186904 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:03:57.186907 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:03:57.187439 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:03:57.187807 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:03:57.187812 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:03:57.187822 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:03:57.187981 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:03:57.188540 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:03:57.188545 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:03:57.188787 (+ 242us) client_negotiation.cc:769] Sending connection context
0808 04:03:57.188981 (+ 194us) client_negotiation.cc:240] Negotiation successful
0808 04:03:57.189305 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":492,"threads_started":1}
I20260808 04:03:57.190325 7274 heartbeater.cc:347] Connected to a master server at 127.5.83.62:45699
I20260808 04:03:57.190536 7274 heartbeater.cc:464] Registering TS with master...
I20260808 04:03:57.191298 7274 heartbeater.cc:511] Master 127.5.83.62:45699 requested a full tablet report, sending...
I20260808 04:03:57.193434 6986 ts_manager.cc:194] Registered new tserver with Master: 5fcb22667c684fa495415f7f391266c1 (127.5.83.1:42131)
I20260808 04:03:57.194797 6986 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.1@KRBTEST.COM'} at 127.5.83.1:51845
I20260808 04:03:57.197552 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 7084
I20260808 04:03:57.197747 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/minicluster-data/ts-0/wal/instance
I20260808 04:03:57.198342 5452 external_mini_cluster.cc:1428] Setting key ca23f407bfbff382f40e43508b0d4f54
WARNING: no policy specified for kudu/127.5.83.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.2.keytab.
Entry for principal kudu/127.5.83.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.2.keytab.
I20260808 04:03:57.239303 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.2:0
--local_ip_for_outbound_sockets=127.5.83.2
--webserver_interface=127.5.83.2
--webserver_port=0
--tserver_master_addrs=127.5.83.62:45699
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.2.keytab
--principal=kudu/127.5.83.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:37307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:03:57.548655 7279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:03:57.548930 7279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:03:57.549068 7279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:03:57.556012 7279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:03:57.556118 7279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:03:57.556239 7279 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 04:03:57.556346 7279 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.2
I20260808 04:03:57.565578 7279 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:37307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.2.keytab
--principal=kudu/127.5.83.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.83.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.83.62:45699
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.2
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:03:57.566704 7279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:03:57.567996 7279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:03:57.580221 7291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:03:57.580188 7287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:03:57.580264 7288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:03:57.581535 7279 server_base.cc:1034] running on GCE node
I20260808 04:03:57.582113 7279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:03:57.582940 7279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:03:57.584147 7279 hybrid_clock.cc:648] HybridClock initialized: now 1786161837584085 us; error 35 us; skew 500 ppm
I20260808 04:03:57.584421 7279 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:03:57 dist-test-slave-1l3l krb5kdc[6923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161837, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:03:57.591212 7279 init.cc:377] Logged in from keytab as kudu/127.5.83.2@KRBTEST.COM (short username kudu)
I20260808 04:03:57.592646 7279 webserver.cc:460] Webserver started at http://127.5.83.2:36027/ using document root <none> and password file <none>
I20260808 04:03:57.593309 7279 fs_manager.cc:359] Metadata directory not provided
I20260808 04:03:57.593410 7279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:03:57.593798 7279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:03:57.597559 7279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/minicluster-data/ts-1/data/instance:
uuid: "61a56ad7bdac46b199107e364bd82d4e"
format_stamp: "Formatted at 2026-08-08 04:03:57 on dist-test-slave-1l3l"
server_key: "a35e046b7c825f5235d1af4f538ec2ec"
server_key_iv: "4ef36c853cec13449aef45cdafc60b0d"
server_key_version: "encryptionkey@0"
I20260808 04:03:57.598518 7279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/minicluster-data/ts-1/wal/instance:
uuid: "61a56ad7bdac46b199107e364bd82d4e"
format_stamp: "Formatted at 2026-08-08 04:03:57 on dist-test-slave-1l3l"
server_key: "a35e046b7c825f5235d1af4f538ec2ec"
server_key_iv: "4ef36c853cec13449aef45cdafc60b0d"
server_key_version: "encryptionkey@0"
I20260808 04:03:57.604698 7279 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260808 04:03:57.608810 7300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:03:57.610206 7279 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 04:03:57.610364 7279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/minicluster-data/ts-1/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/minicluster-data/ts-1/wal
uuid: "61a56ad7bdac46b199107e364bd82d4e"
format_stamp: "Formatted at 2026-08-08 04:03:57 on dist-test-slave-1l3l"
server_key: "a35e046b7c825f5235d1af4f538ec2ec"
server_key_iv: "4ef36c853cec13449aef45cdafc60b0d"
server_key_version: "encryptionkey@0"
I20260808 04:03:57.610587 7279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:03:57.622777 7279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:03:57.626729 7279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:03:57.626952 7279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:03:57.629539 7279 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:03:57.629617 7279 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:03:57.629750 7279 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:03:57.629856 7279 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Aug 08 04:03:57 dist-test-slave-1l3l krb5kdc[6923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161837, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.2@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:03:57.651594 7120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:57.630004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.2:56239 (local address 127.5.83.62:45699)
0808 04:03:57.630196 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:57.630202 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:57.631425 (+ 1223us) server_negotiation.cc:407] Connection header received
0808 04:03:57.633580 (+ 2155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:57.633586 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:57.633682 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:57.633801 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:57.635891 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:57.636989 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:57.639437 (+ 2448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:57.639955 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:57.640087 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:57.645062 (+ 4975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:57.645069 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:57.645074 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:03:57.645115 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:57.647911 (+ 2796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:57.648860 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:57.648866 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:57.648869 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:57.649024 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:57.649549 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:57.649555 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:57.649559 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:57.649994 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:57.650154 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:57.650948 (+ 794us) server_negotiation.cc:299] Negotiation successful
0808 04:03:57.651216 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 04:03:57.652417 7310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:57.630458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:45699 (local address 127.5.83.2:56239)
0808 04:03:57.631245 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:57.631299 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0808 04:03:57.633306 (+ 2007us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:03:57.634077 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:03:57.634088 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:03:57.634809 (+ 721us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:03:57.635658 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:57.635687 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:57.637185 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:57.637190 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:57.639293 (+ 2103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:57.639305 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:57.640307 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:57.640319 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:57.640629 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:57.641337 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:03:57.641371 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:03:57.644773 (+ 3402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:03:57.648065 (+ 3292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:03:57.648072 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:03:57.648085 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:03:57.648691 (+ 606us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:03:57.649122 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:03:57.649127 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:03:57.649131 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:03:57.649399 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:03:57.650116 (+ 717us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:03:57.650121 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:03:57.650463 (+ 342us) client_negotiation.cc:769] Sending connection context
0808 04:03:57.650799 (+ 336us) client_negotiation.cc:240] Negotiation successful
0808 04:03:57.651223 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":364,"threads_started":1}
I20260808 04:03:57.671041 7120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:57.658092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.2:43427 (local address 127.5.83.62:45699)
0808 04:03:57.658269 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:57.658275 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:57.658772 (+ 497us) server_negotiation.cc:407] Connection header received
0808 04:03:57.658893 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:57.658899 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:57.658976 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:57.659079 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:57.659912 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:57.660895 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:57.662769 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:57.663174 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:57.663280 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:57.665465 (+ 2185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:57.665470 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:57.665473 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:03:57.665503 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:57.668408 (+ 2905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:57.669247 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:57.669251 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:57.669253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:57.669358 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:57.669717 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:57.669722 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:57.669725 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:57.670059 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:57.670175 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:57.670580 (+ 405us) server_negotiation.cc:299] Negotiation successful
0808 04:03:57.670777 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 04:03:57.671370 7407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:57.658076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:45699 (local address 127.5.83.2:43427)
0808 04:03:57.658686 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:57.658702 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:03:57.658805 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:03:57.659087 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:03:57.659089 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:03:57.659470 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:03:57.659782 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:57.659792 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:57.661048 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:57.661052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:57.662641 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:57.662653 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:57.663346 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:57.663352 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:57.663502 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:57.664112 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:03:57.664131 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:03:57.665178 (+ 1047us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:03:57.668562 (+ 3384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:03:57.668568 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:03:57.668572 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:03:57.669138 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:03:57.669456 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:03:57.669461 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:03:57.669465 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:03:57.669613 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:03:57.670151 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:03:57.670156 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:03:57.670354 (+ 198us) client_negotiation.cc:769] Sending connection context
0808 04:03:57.670503 (+ 149us) client_negotiation.cc:240] Negotiation successful
0808 04:03:57.670869 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":319,"threads_started":1}
I20260808 04:03:57.678996 7279 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.2:40595
I20260808 04:03:57.679034 7463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.2:40595 every 8 connection(s)
I20260808 04:03:57.680271 7279 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
I20260808 04:03:57.682209 7464 heartbeater.cc:347] Connected to a master server at 127.5.83.62:45699
I20260808 04:03:57.682410 7464 heartbeater.cc:464] Registering TS with master...
I20260808 04:03:57.682967 7464 heartbeater.cc:511] Master 127.5.83.62:45699 requested a full tablet report, sending...
I20260808 04:03:57.684198 6986 ts_manager.cc:194] Registered new tserver with Master: 61a56ad7bdac46b199107e364bd82d4e (127.5.83.2:40595)
I20260808 04:03:57.684950 6986 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.2@KRBTEST.COM'} at 127.5.83.2:56239
I20260808 04:03:57.686007 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 7279
I20260808 04:03:57.686193 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/minicluster-data/ts-1/wal/instance
I20260808 04:03:57.686748 5452 external_mini_cluster.cc:1428] Setting key 89742e4156a875781ffb856579a4e8c6
WARNING: no policy specified for kudu/127.5.83.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.3.keytab.
Entry for principal kudu/127.5.83.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.3.keytab.
I20260808 04:03:57.725242 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.3:0
--local_ip_for_outbound_sockets=127.5.83.3
--webserver_interface=127.5.83.3
--webserver_port=0
--tserver_master_addrs=127.5.83.62:45699
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.3.keytab
--principal=kudu/127.5.83.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:37307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:03:58.197314 7274 heartbeater.cc:503] Master 127.5.83.62:45699 was elected leader, sending a full tablet report...
W20260808 04:03:58.214918 7470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:03:58.215489 7470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:03:58.215765 7470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:03:58.228121 7470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:03:58.228292 7470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:03:58.228463 7470 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 04:03:58.228679 7470 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.3
I20260808 04:03:58.246594 7470 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:37307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.3.keytab
--principal=kudu/127.5.83.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.83.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.83.62:45699
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.3
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:03:58.249001 7470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:03:58.251240 7470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:03:58.273000 7477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:03:58.273170 7478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:03:58.273125 7480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:03:58.274710 7470 server_base.cc:1034] running on GCE node
I20260808 04:03:58.275663 7470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:03:58.277099 7470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:03:58.278450 7470 hybrid_clock.cc:648] HybridClock initialized: now 1786161838278363 us; error 61 us; skew 500 ppm
I20260808 04:03:58.278836 7470 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:03:58 dist-test-slave-1l3l krb5kdc[6923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161838, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:03:58.283979 7470 init.cc:377] Logged in from keytab as kudu/127.5.83.3@KRBTEST.COM (short username kudu)
I20260808 04:03:58.286334 7470 webserver.cc:460] Webserver started at http://127.5.83.3:44205/ using document root <none> and password file <none>
I20260808 04:03:58.287412 7470 fs_manager.cc:359] Metadata directory not provided
I20260808 04:03:58.287580 7470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:03:58.288136 7470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:03:58.292975 7470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/minicluster-data/ts-2/data/instance:
uuid: "a46882de0a954f6b93c5449d6e09f340"
format_stamp: "Formatted at 2026-08-08 04:03:58 on dist-test-slave-1l3l"
server_key: "ed886fd697fb1e451a911359be9df695"
server_key_iv: "8ac662df99fc6f3b83a81c0b4878dcde"
server_key_version: "encryptionkey@0"
I20260808 04:03:58.294224 7470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/minicluster-data/ts-2/wal/instance:
uuid: "a46882de0a954f6b93c5449d6e09f340"
format_stamp: "Formatted at 2026-08-08 04:03:58 on dist-test-slave-1l3l"
server_key: "ed886fd697fb1e451a911359be9df695"
server_key_iv: "8ac662df99fc6f3b83a81c0b4878dcde"
server_key_version: "encryptionkey@0"
I20260808 04:03:58.302804 7470 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.009s sys 0.000s
I20260808 04:03:58.308254 7491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:03:58.310338 7470 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260808 04:03:58.310554 7470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/minicluster-data/ts-2/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/minicluster-data/ts-2/wal
uuid: "a46882de0a954f6b93c5449d6e09f340"
format_stamp: "Formatted at 2026-08-08 04:03:58 on dist-test-slave-1l3l"
server_key: "ed886fd697fb1e451a911359be9df695"
server_key_iv: "8ac662df99fc6f3b83a81c0b4878dcde"
server_key_version: "encryptionkey@0"
I20260808 04:03:58.310823 7470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:03:58.335490 7470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:03:58.342396 7470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:03:58.342829 7470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:03:58.347482 7470 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:03:58.347649 7470 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:03:58.347929 7470 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:03:58.348112 7470 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 04:03:58 dist-test-slave-1l3l krb5kdc[6923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161838, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.3@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:03:58.391592 7502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:58.348131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.3:48573 (local address 127.5.83.62:45699)
0808 04:03:58.349703 (+ 1572us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:58.349713 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:58.351188 (+ 1475us) server_negotiation.cc:407] Connection header received
0808 04:03:58.354915 (+ 3727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:58.354933 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:58.355115 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:58.355429 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:58.359243 (+ 3814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:58.361242 (+ 1999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:58.366557 (+ 5315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:58.367939 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:58.368216 (+ 277us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:58.377758 (+ 9542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:58.377777 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:58.377782 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:03:58.377873 (+ 91us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:58.384258 (+ 6385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:58.386284 (+ 2026us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:58.386356 (+ 72us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:58.386360 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:58.386680 (+ 320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:58.387753 (+ 1073us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:58.387770 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:58.387773 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:58.388664 (+ 891us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:58.388927 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:58.390246 (+ 1319us) server_negotiation.cc:299] Negotiation successful
0808 04:03:58.390669 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1218,"thread_start_us":1032,"threads_started":1}
I20260808 04:03:58.394171 7504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:58.349137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:45699 (local address 127.5.83.3:48573)
0808 04:03:58.350895 (+ 1758us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:58.350984 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0808 04:03:58.354404 (+ 3420us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:03:58.355928 (+ 1524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:03:58.355952 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:03:58.357352 (+ 1400us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:03:58.358891 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:58.358948 (+ 57us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:58.361643 (+ 2695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:58.361660 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:58.366300 (+ 4640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:58.366335 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:58.368456 (+ 2121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:58.368478 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:58.368968 (+ 490us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:58.370429 (+ 1461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:03:58.370537 (+ 108us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:03:58.377175 (+ 6638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:03:58.384510 (+ 7335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:03:58.384523 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:03:58.384553 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:03:58.386020 (+ 1467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:03:58.386916 (+ 896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:03:58.386922 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:03:58.386925 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:03:58.387465 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:03:58.388943 (+ 1478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:03:58.388953 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:03:58.389583 (+ 630us) client_negotiation.cc:769] Sending connection context
0808 04:03:58.389992 (+ 409us) client_negotiation.cc:240] Negotiation successful
0808 04:03:58.390698 (+ 706us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1215,"thread_start_us":642,"threads_started":1}
I20260808 04:03:58.419153 7470 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.3:39863
I20260808 04:03:58.419227 7663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.3:39863 every 8 connection(s)
I20260808 04:03:58.422050 7470 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
I20260808 04:03:58.425953 7665 heartbeater.cc:347] Connected to a master server at 127.5.83.62:45699
I20260808 04:03:58.426348 7665 heartbeater.cc:464] Registering TS with master...
I20260808 04:03:58.427663 7665 heartbeater.cc:511] Master 127.5.83.62:45699 requested a full tablet report, sending...
I20260808 04:03:58.427744 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 7470
I20260808 04:03:58.428133 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/minicluster-data/ts-2/wal/instance
I20260808 04:03:58.429118 5452 external_mini_cluster.cc:1428] Setting key c7a245fcbdd1346f30bb397394b7dcbf
I20260808 04:03:58.429930 6986 ts_manager.cc:194] Registered new tserver with Master: a46882de0a954f6b93c5449d6e09f340 (127.5.83.3:39863)
I20260808 04:03:58.431289 6986 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.3@KRBTEST.COM'} at 127.5.83.3:48573
I20260808 04:03:58.435042 7642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:58.409138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:45699 (local address 127.5.83.3:54653)
0808 04:03:58.410198 (+ 1060us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:58.410240 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 04:03:58.410490 (+ 250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:03:58.411441 (+ 951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:03:58.411459 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:03:58.412677 (+ 1218us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:03:58.413376 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:58.413412 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:58.415974 (+ 2562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:58.416005 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:58.418513 (+ 2508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:58.418560 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:58.420045 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:58.420054 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:58.420466 (+ 412us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:58.421496 (+ 1030us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:03:58.421548 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:03:58.423799 (+ 2251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:03:58.430425 (+ 6626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:03:58.430430 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:03:58.430444 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:03:58.431029 (+ 585us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:03:58.431656 (+ 627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:03:58.431661 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:03:58.431664 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:03:58.431929 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:03:58.433178 (+ 1249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:03:58.433181 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:03:58.433625 (+ 444us) client_negotiation.cc:769] Sending connection context
0808 04:03:58.433976 (+ 351us) client_negotiation.cc:240] Negotiation successful
0808 04:03:58.434497 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":795,"thread_start_us":536,"threads_started":1}
I20260808 04:03:58.436813 5452 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:03:58.437248 7502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:58.409127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.3:54653 (local address 127.5.83.62:45699)
0808 04:03:58.409480 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:58.409497 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:58.410369 (+ 872us) server_negotiation.cc:407] Connection header received
0808 04:03:58.410803 (+ 434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:58.410811 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:58.410938 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:58.411068 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:58.413641 (+ 2573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:58.415742 (+ 2101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:58.418831 (+ 3089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:58.419732 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:58.419920 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:58.424199 (+ 4279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:58.424208 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:58.424224 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:03:58.424301 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:58.430188 (+ 5887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:58.431239 (+ 1051us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:58.431246 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:58.431249 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:58.431476 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:58.432171 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:58.432178 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:58.432180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:58.432794 (+ 614us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:58.433064 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:58.436104 (+ 3040us) server_negotiation.cc:299] Negotiation successful
0808 04:03:58.436600 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
Aug 08 04:03:58 dist-test-slave-1l3l krb5kdc[6923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161838, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 04:03:58 dist-test-slave-1l3l krb5kdc[6923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161838, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:03:58.510519 7502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:58.487871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44018 (local address 127.5.83.62:45699)
0808 04:03:58.488235 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:58.488253 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:58.488581 (+ 328us) server_negotiation.cc:407] Connection header received
0808 04:03:58.488834 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:58.488849 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:58.488968 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:58.489100 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:58.491540 (+ 2440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:58.492903 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:58.495026 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:58.495687 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:58.496037 (+ 350us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:58.502455 (+ 6418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:58.502461 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:58.502473 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:03:58.502532 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:58.506550 (+ 4018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:58.507658 (+ 1108us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:58.507663 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:58.507665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:58.507958 (+ 293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:58.508545 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:58.508549 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:58.508551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:58.508945 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:58.509259 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:58.509671 (+ 412us) server_negotiation.cc:299] Negotiation successful
0808 04:03:58.509928 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260808 04:03:58.517199 6986 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44018:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:03:58.529595 6986 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:03:58.572096 7690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:58.551044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56462 (local address 127.5.83.3:39863)
0808 04:03:58.552746 (+ 1702us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:58.552769 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:58.553758 (+ 989us) server_negotiation.cc:407] Connection header received
0808 04:03:58.554598 (+ 840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:58.554621 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:58.555131 (+ 510us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:58.555469 (+ 338us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:03:58.557796 (+ 2327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:58.560905 (+ 3109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:58.566164 (+ 5259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:58.568933 (+ 2769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:58.569134 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:58.569654 (+ 520us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:58.571074 (+ 1420us) server_negotiation.cc:299] Negotiation successful
0808 04:03:58.571388 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1416,"thread_start_us":1064,"threads_started":1}
I20260808 04:03:58.572786 7688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:58.550903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:42131 (local address 127.0.0.1:39184)
0808 04:03:58.554446 (+ 3543us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:58.554484 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0808 04:03:58.554834 (+ 350us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:03:58.556664 (+ 1830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:03:58.556680 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:03:58.556727 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:03:58.557708 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:58.557731 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:58.561289 (+ 3558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:58.561298 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:58.565700 (+ 4402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:58.565732 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:58.570867 (+ 5135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:58.570893 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:58.571338 (+ 445us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:58.571411 (+ 73us) client_negotiation.cc:769] Sending connection context
0808 04:03:58.571562 (+ 151us) client_negotiation.cc:240] Negotiation successful
0808 04:03:58.572005 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3311,"mutex_wait_us":21,"thread_start_us":2983,"threads_started":1}
I20260808 04:03:58.573943 7691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:58.551875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39184 (local address 127.5.83.1:42131)
0808 04:03:58.553617 (+ 1742us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:58.553642 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:58.554659 (+ 1017us) server_negotiation.cc:407] Connection header received
0808 04:03:58.555326 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:58.555346 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:58.555833 (+ 487us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:58.556297 (+ 464us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:03:58.558073 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:58.560960 (+ 2887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:58.566021 (+ 5061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:58.568933 (+ 2912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:58.569134 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:58.569611 (+ 477us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:58.573236 (+ 3625us) server_negotiation.cc:299] Negotiation successful
0808 04:03:58.573519 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1362,"thread_start_us":1089,"threads_started":1}
I20260808 04:03:58.574344 7687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:58.550365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.3:39863 (local address 127.0.0.1:56462)
0808 04:03:58.553352 (+ 2987us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:58.553528 (+ 176us) client_negotiation.cc:174] Beginning negotiation
0808 04:03:58.554027 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:03:58.556297 (+ 2270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:03:58.556319 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:03:58.556388 (+ 69us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:03:58.557317 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:58.557388 (+ 71us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:58.562095 (+ 4707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:58.562112 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:58.565872 (+ 3760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:58.565889 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:58.570032 (+ 4143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:58.570042 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:58.570535 (+ 493us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:58.570655 (+ 120us) client_negotiation.cc:769] Sending connection context
0808 04:03:58.572882 (+ 2227us) client_negotiation.cc:240] Negotiation successful
0808 04:03:58.573492 (+ 610us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2539,"mutex_wait_us":58,"thread_start_us":2339,"threads_started":1}
I20260808 04:03:58.580655 7562 tablet_service.cc:1467] Processing CreateTablet for tablet 626d79b7b09940d4af54e73839f09a25 (DEFAULT_TABLE table=test-table [id=61904e4f42c44736965c5dd7b4ce0616]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:03:58.580889 7179 tablet_service.cc:1467] Processing CreateTablet for tablet 626d79b7b09940d4af54e73839f09a25 (DEFAULT_TABLE table=test-table [id=61904e4f42c44736965c5dd7b4ce0616]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:03:58.582973 7562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 626d79b7b09940d4af54e73839f09a25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:03:58.583746 7179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 626d79b7b09940d4af54e73839f09a25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:03:58.589464 7693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:58.554635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.2:40595 (local address 127.0.0.1:56342)
0808 04:03:58.573285 (+ 18650us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:58.573318 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0808 04:03:58.573516 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:03:58.575050 (+ 1534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:03:58.575067 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:03:58.575117 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:03:58.575788 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:58.575815 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:58.578951 (+ 3136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:58.578958 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:58.583235 (+ 4277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:58.583293 (+ 58us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:58.587922 (+ 4629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:58.587940 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:58.588397 (+ 457us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:58.588470 (+ 73us) client_negotiation.cc:769] Sending connection context
0808 04:03:58.588617 (+ 147us) client_negotiation.cc:240] Negotiation successful
0808 04:03:58.588864 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18322,"thread_start_us":18080,"threads_started":1}
I20260808 04:03:58.590595 7692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:58.554666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56342 (local address 127.5.83.2:40595)
0808 04:03:58.570630 (+ 15964us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:58.570641 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:58.573582 (+ 2941us) server_negotiation.cc:407] Connection header received
0808 04:03:58.573778 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:58.573788 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:58.574296 (+ 508us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:58.574652 (+ 356us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:03:58.576137 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:58.578536 (+ 2399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:58.583671 (+ 5135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:58.586935 (+ 3264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:58.587186 (+ 251us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:58.587666 (+ 480us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:58.589819 (+ 2153us) server_negotiation.cc:299] Negotiation successful
0808 04:03:58.590103 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15655,"thread_start_us":15492,"threads_started":1}
I20260808 04:03:58.597337 7369 tablet_service.cc:1467] Processing CreateTablet for tablet 626d79b7b09940d4af54e73839f09a25 (DEFAULT_TABLE table=test-table [id=61904e4f42c44736965c5dd7b4ce0616]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:03:58.599932 7369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 626d79b7b09940d4af54e73839f09a25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:03:58.615636 7694 tablet_bootstrap.cc:492] T 626d79b7b09940d4af54e73839f09a25 P a46882de0a954f6b93c5449d6e09f340: Bootstrap starting.
I20260808 04:03:58.625295 7695 tablet_bootstrap.cc:492] T 626d79b7b09940d4af54e73839f09a25 P 5fcb22667c684fa495415f7f391266c1: Bootstrap starting.
I20260808 04:03:58.627370 7694 tablet_bootstrap.cc:654] T 626d79b7b09940d4af54e73839f09a25 P a46882de0a954f6b93c5449d6e09f340: Neither blocks nor log segments found. Creating new log.
I20260808 04:03:58.629882 7694 log.cc:824] T 626d79b7b09940d4af54e73839f09a25 P a46882de0a954f6b93c5449d6e09f340: Log is configured to *not* fsync() on all Append() calls
I20260808 04:03:58.631599 7697 tablet_bootstrap.cc:492] T 626d79b7b09940d4af54e73839f09a25 P 61a56ad7bdac46b199107e364bd82d4e: Bootstrap starting.
I20260808 04:03:58.632445 7695 tablet_bootstrap.cc:654] T 626d79b7b09940d4af54e73839f09a25 P 5fcb22667c684fa495415f7f391266c1: Neither blocks nor log segments found. Creating new log.
I20260808 04:03:58.635070 7695 log.cc:824] T 626d79b7b09940d4af54e73839f09a25 P 5fcb22667c684fa495415f7f391266c1: Log is configured to *not* fsync() on all Append() calls
I20260808 04:03:58.637862 7697 tablet_bootstrap.cc:654] T 626d79b7b09940d4af54e73839f09a25 P 61a56ad7bdac46b199107e364bd82d4e: Neither blocks nor log segments found. Creating new log.
I20260808 04:03:58.640064 7694 tablet_bootstrap.cc:492] T 626d79b7b09940d4af54e73839f09a25 P a46882de0a954f6b93c5449d6e09f340: No bootstrap required, opened a new log
I20260808 04:03:58.640067 7697 log.cc:824] T 626d79b7b09940d4af54e73839f09a25 P 61a56ad7bdac46b199107e364bd82d4e: Log is configured to *not* fsync() on all Append() calls
I20260808 04:03:58.640394 7695 tablet_bootstrap.cc:492] T 626d79b7b09940d4af54e73839f09a25 P 5fcb22667c684fa495415f7f391266c1: No bootstrap required, opened a new log
I20260808 04:03:58.640662 7694 ts_tablet_manager.cc:1397] T 626d79b7b09940d4af54e73839f09a25 P a46882de0a954f6b93c5449d6e09f340: Time spent bootstrapping tablet: real 0.026s user 0.006s sys 0.011s
I20260808 04:03:58.640803 7695 ts_tablet_manager.cc:1397] T 626d79b7b09940d4af54e73839f09a25 P 5fcb22667c684fa495415f7f391266c1: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.005s
I20260808 04:03:58.644169 7697 tablet_bootstrap.cc:492] T 626d79b7b09940d4af54e73839f09a25 P 61a56ad7bdac46b199107e364bd82d4e: No bootstrap required, opened a new log
I20260808 04:03:58.644567 7697 ts_tablet_manager.cc:1397] T 626d79b7b09940d4af54e73839f09a25 P 61a56ad7bdac46b199107e364bd82d4e: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.004s
I20260808 04:03:58.656980 7697 raft_consensus.cc:348] T 626d79b7b09940d4af54e73839f09a25 P 61a56ad7bdac46b199107e364bd82d4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46882de0a954f6b93c5449d6e09f340" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 39863 } } peers { permanent_uuid: "5fcb22667c684fa495415f7f391266c1" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42131 } } peers { permanent_uuid: "61a56ad7bdac46b199107e364bd82d4e" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 40595 } }
I20260808 04:03:58.657620 7697 raft_consensus.cc:374] T 626d79b7b09940d4af54e73839f09a25 P 61a56ad7bdac46b199107e364bd82d4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:03:58.657785 7697 raft_consensus.cc:729] T 626d79b7b09940d4af54e73839f09a25 P 61a56ad7bdac46b199107e364bd82d4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61a56ad7bdac46b199107e364bd82d4e, State: Initialized, Role: FOLLOWER
I20260808 04:03:58.659047 7697 consensus_queue.cc:260] T 626d79b7b09940d4af54e73839f09a25 P 61a56ad7bdac46b199107e364bd82d4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46882de0a954f6b93c5449d6e09f340" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 39863 } } peers { permanent_uuid: "5fcb22667c684fa495415f7f391266c1" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42131 } } peers { permanent_uuid: "61a56ad7bdac46b199107e364bd82d4e" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 40595 } }
I20260808 04:03:58.662205 7464 heartbeater.cc:503] Master 127.5.83.62:45699 was elected leader, sending a full tablet report...
I20260808 04:03:58.662525 7695 raft_consensus.cc:348] T 626d79b7b09940d4af54e73839f09a25 P 5fcb22667c684fa495415f7f391266c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46882de0a954f6b93c5449d6e09f340" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 39863 } } peers { permanent_uuid: "5fcb22667c684fa495415f7f391266c1" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42131 } } peers { permanent_uuid: "61a56ad7bdac46b199107e364bd82d4e" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 40595 } }
I20260808 04:03:58.662523 7694 raft_consensus.cc:348] T 626d79b7b09940d4af54e73839f09a25 P a46882de0a954f6b93c5449d6e09f340 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46882de0a954f6b93c5449d6e09f340" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 39863 } } peers { permanent_uuid: "5fcb22667c684fa495415f7f391266c1" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42131 } } peers { permanent_uuid: "61a56ad7bdac46b199107e364bd82d4e" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 40595 } }
I20260808 04:03:58.663509 7694 raft_consensus.cc:374] T 626d79b7b09940d4af54e73839f09a25 P a46882de0a954f6b93c5449d6e09f340 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:03:58.663455 7695 raft_consensus.cc:374] T 626d79b7b09940d4af54e73839f09a25 P 5fcb22667c684fa495415f7f391266c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:03:58.663553 7697 ts_tablet_manager.cc:1428] T 626d79b7b09940d4af54e73839f09a25 P 61a56ad7bdac46b199107e364bd82d4e: Time spent starting tablet: real 0.019s user 0.017s sys 0.000s
I20260808 04:03:58.663681 7694 raft_consensus.cc:729] T 626d79b7b09940d4af54e73839f09a25 P a46882de0a954f6b93c5449d6e09f340 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a46882de0a954f6b93c5449d6e09f340, State: Initialized, Role: FOLLOWER
I20260808 04:03:58.663796 7695 raft_consensus.cc:729] T 626d79b7b09940d4af54e73839f09a25 P 5fcb22667c684fa495415f7f391266c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcb22667c684fa495415f7f391266c1, State: Initialized, Role: FOLLOWER
I20260808 04:03:58.664927 7695 consensus_queue.cc:260] T 626d79b7b09940d4af54e73839f09a25 P 5fcb22667c684fa495415f7f391266c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46882de0a954f6b93c5449d6e09f340" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 39863 } } peers { permanent_uuid: "5fcb22667c684fa495415f7f391266c1" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42131 } } peers { permanent_uuid: "61a56ad7bdac46b199107e364bd82d4e" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 40595 } }
I20260808 04:03:58.664915 7694 consensus_queue.cc:260] T 626d79b7b09940d4af54e73839f09a25 P a46882de0a954f6b93c5449d6e09f340 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46882de0a954f6b93c5449d6e09f340" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 39863 } } peers { permanent_uuid: "5fcb22667c684fa495415f7f391266c1" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42131 } } peers { permanent_uuid: "61a56ad7bdac46b199107e364bd82d4e" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 40595 } }
I20260808 04:03:58.666908 7665 heartbeater.cc:503] Master 127.5.83.62:45699 was elected leader, sending a full tablet report...
I20260808 04:03:58.668267 7694 ts_tablet_manager.cc:1428] T 626d79b7b09940d4af54e73839f09a25 P a46882de0a954f6b93c5449d6e09f340: Time spent starting tablet: real 0.027s user 0.020s sys 0.007s
I20260808 04:03:58.670293 7695 ts_tablet_manager.cc:1428] T 626d79b7b09940d4af54e73839f09a25 P 5fcb22667c684fa495415f7f391266c1: Time spent starting tablet: real 0.029s user 0.028s sys 0.000s
W20260808 04:03:58.675182 7666 tablet.cc:2369] T 626d79b7b09940d4af54e73839f09a25 P a46882de0a954f6b93c5449d6e09f340: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 04:03:58.682366 7465 tablet.cc:2369] T 626d79b7b09940d4af54e73839f09a25 P 61a56ad7bdac46b199107e364bd82d4e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 04:03:58.690614 7275 tablet.cc:2369] T 626d79b7b09940d4af54e73839f09a25 P 5fcb22667c684fa495415f7f391266c1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:03:58.709828 7703 raft_consensus.cc:482] T 626d79b7b09940d4af54e73839f09a25 P 5fcb22667c684fa495415f7f391266c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:03:58.710132 7703 raft_consensus.cc:504] T 626d79b7b09940d4af54e73839f09a25 P 5fcb22667c684fa495415f7f391266c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46882de0a954f6b93c5449d6e09f340" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 39863 } } peers { permanent_uuid: "5fcb22667c684fa495415f7f391266c1" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42131 } } peers { permanent_uuid: "61a56ad7bdac46b199107e364bd82d4e" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 40595 } }
I20260808 04:03:58.711673 7703 leader_election.cc:288] T 626d79b7b09940d4af54e73839f09a25 P 5fcb22667c684fa495415f7f391266c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a46882de0a954f6b93c5449d6e09f340 (127.5.83.3:39863), 61a56ad7bdac46b199107e364bd82d4e (127.5.83.2:40595)
I20260808 04:03:58.713037 7702 raft_consensus.cc:482] T 626d79b7b09940d4af54e73839f09a25 P 61a56ad7bdac46b199107e364bd82d4e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:03:58.713336 7702 raft_consensus.cc:504] T 626d79b7b09940d4af54e73839f09a25 P 61a56ad7bdac46b199107e364bd82d4e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46882de0a954f6b93c5449d6e09f340" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 39863 } } peers { permanent_uuid: "5fcb22667c684fa495415f7f391266c1" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42131 } } peers { permanent_uuid: "61a56ad7bdac46b199107e364bd82d4e" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 40595 } }
I20260808 04:03:58.717291 7702 leader_election.cc:288] T 626d79b7b09940d4af54e73839f09a25 P 61a56ad7bdac46b199107e364bd82d4e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a46882de0a954f6b93c5449d6e09f340 (127.5.83.3:39863), 5fcb22667c684fa495415f7f391266c1 (127.5.83.1:42131)
I20260808 04:03:58.723690 7690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:58.712166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.1:58753 (local address 127.5.83.3:39863)
0808 04:03:58.712337 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:58.712341 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:58.712917 (+ 576us) server_negotiation.cc:407] Connection header received
0808 04:03:58.713232 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:58.713239 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:58.713322 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:58.713385 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:03:58.714397 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:58.715570 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:58.720535 (+ 4965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:58.721725 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:58.721817 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:58.721930 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:58.723220 (+ 1290us) server_negotiation.cc:299] Negotiation successful
0808 04:03:58.723353 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 04:03:58.724402 7692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:58.713139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.1:54043 (local address 127.5.83.2:40595)
0808 04:03:58.713328 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:58.713334 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:58.717108 (+ 3774us) server_negotiation.cc:407] Connection header received
0808 04:03:58.717363 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:58.717367 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:58.717432 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:58.717490 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:03:58.718427 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:58.719616 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:58.721996 (+ 2380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:58.723152 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:58.723276 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:58.723370 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:58.723941 (+ 571us) server_negotiation.cc:299] Negotiation successful
0808 04:03:58.724138 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 04:03:58.724514 7705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:58.712091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.3:39863 (local address 127.5.83.1:58753)
0808 04:03:58.712806 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:58.712825 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 04:03:58.713046 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:03:58.713606 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:03:58.713611 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:03:58.713640 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:03:58.714207 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:58.714226 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:58.717792 (+ 3566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:58.717797 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:58.720362 (+ 2565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:58.720376 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:58.722111 (+ 1735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:58.722117 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:58.723015 (+ 898us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:58.723057 (+ 42us) client_negotiation.cc:769] Sending connection context
0808 04:03:58.723947 (+ 890us) client_negotiation.cc:240] Negotiation successful
0808 04:03:58.724127 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"mutex_wait_us":3,"thread_start_us":385,"threads_started":1}
I20260808 04:03:58.725099 7691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:58.715470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.2:33063 (local address 127.5.83.1:42131)
0808 04:03:58.715630 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:58.715635 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:58.717368 (+ 1733us) server_negotiation.cc:407] Connection header received
0808 04:03:58.717646 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:58.717651 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:58.717731 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:58.717797 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:03:58.718605 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:58.719759 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:58.722263 (+ 2504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:58.723356 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:58.723481 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:58.723579 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:58.724345 (+ 766us) server_negotiation.cc:299] Negotiation successful
0808 04:03:58.724466 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":47,"spinlock_wait_cycles":1280}
I20260808 04:03:58.725133 7709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:58.715361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.3:39863 (local address 127.5.83.2:58337)
0808 04:03:58.716249 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:58.716272 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 04:03:58.716401 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:03:58.718201 (+ 1800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:03:58.718206 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:03:58.718235 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:03:58.718769 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:58.718783 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:58.720531 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:58.720537 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:58.722602 (+ 2065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:58.722617 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:58.724478 (+ 1861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:58.724485 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:58.724641 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:58.724672 (+ 31us) client_negotiation.cc:769] Sending connection context
0808 04:03:58.724736 (+ 64us) client_negotiation.cc:240] Negotiation successful
0808 04:03:58.724856 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":730,"thread_start_us":823,"threads_started":1}
I20260808 04:03:58.725009 7707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:58.712883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.2:40595 (local address 127.5.83.1:54043)
0808 04:03:58.717011 (+ 4128us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:58.717030 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 04:03:58.717205 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:03:58.717657 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:03:58.717661 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:03:58.717688 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:03:58.718241 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:58.718255 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:58.719799 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:58.719803 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:58.721805 (+ 2002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:58.721819 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:58.723526 (+ 1707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:58.723533 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:58.723691 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:58.723742 (+ 51us) client_negotiation.cc:769] Sending connection context
0808 04:03:58.724557 (+ 815us) client_negotiation.cc:240] Negotiation successful
0808 04:03:58.724685 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4001,"thread_start_us":3827,"threads_started":1}
I20260808 04:03:58.725660 7710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:58.715541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:42131 (local address 127.5.83.2:33063)
0808 04:03:58.717297 (+ 1756us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:58.717312 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:03:58.717455 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:03:58.718009 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:03:58.718014 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:03:58.718039 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:03:58.718410 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:58.718424 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:58.719916 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:58.719921 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:58.722087 (+ 2166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:58.722101 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:58.723740 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:58.723747 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:58.724154 (+ 407us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:58.724189 (+ 35us) client_negotiation.cc:769] Sending connection context
0808 04:03:58.725166 (+ 977us) client_negotiation.cc:240] Negotiation successful
0808 04:03:58.725293 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1626,"mutex_wait_us":1,"thread_start_us":1460,"threads_started":1}
I20260808 04:03:58.725641 7593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "626d79b7b09940d4af54e73839f09a25" candidate_uuid: "5fcb22667c684fa495415f7f391266c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a46882de0a954f6b93c5449d6e09f340" is_pre_election: true
I20260808 04:03:58.726042 7714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:58.715949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.2:58337 (local address 127.5.83.3:39863)
0808 04:03:58.717874 (+ 1925us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:58.717879 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:58.717900 (+ 21us) server_negotiation.cc:407] Connection header received
0808 04:03:58.717957 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:58.717962 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:58.718029 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:58.718910 (+ 881us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:03:58.719185 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:58.720364 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:58.722743 (+ 2379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:58.724220 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:58.725340 (+ 1120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:58.725472 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:58.725522 (+ 50us) server_negotiation.cc:299] Negotiation successful
0808 04:03:58.725641 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1798,"thread_start_us":1644,"threads_started":1}
I20260808 04:03:58.726267 7593 raft_consensus.cc:2454] T 626d79b7b09940d4af54e73839f09a25 P a46882de0a954f6b93c5449d6e09f340 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5fcb22667c684fa495415f7f391266c1 in term 0.
I20260808 04:03:58.726485 7204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "626d79b7b09940d4af54e73839f09a25" candidate_uuid: "61a56ad7bdac46b199107e364bd82d4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcb22667c684fa495415f7f391266c1" is_pre_election: true
I20260808 04:03:58.726631 7592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "626d79b7b09940d4af54e73839f09a25" candidate_uuid: "61a56ad7bdac46b199107e364bd82d4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a46882de0a954f6b93c5449d6e09f340" is_pre_election: true
I20260808 04:03:58.726907 7592 raft_consensus.cc:2454] T 626d79b7b09940d4af54e73839f09a25 P a46882de0a954f6b93c5449d6e09f340 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61a56ad7bdac46b199107e364bd82d4e in term 0.
I20260808 04:03:58.726922 7204 raft_consensus.cc:2454] T 626d79b7b09940d4af54e73839f09a25 P 5fcb22667c684fa495415f7f391266c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61a56ad7bdac46b199107e364bd82d4e in term 0.
I20260808 04:03:58.726922 7400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "626d79b7b09940d4af54e73839f09a25" candidate_uuid: "5fcb22667c684fa495415f7f391266c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61a56ad7bdac46b199107e364bd82d4e" is_pre_election: true
I20260808 04:03:58.727031 7113 leader_election.cc:302] T 626d79b7b09940d4af54e73839f09a25 P 5fcb22667c684fa495415f7f391266c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5fcb22667c684fa495415f7f391266c1, a46882de0a954f6b93c5449d6e09f340; no voters:
I20260808 04:03:58.727293 7400 raft_consensus.cc:2454] T 626d79b7b09940d4af54e73839f09a25 P 61a56ad7bdac46b199107e364bd82d4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5fcb22667c684fa495415f7f391266c1 in term 0.
I20260808 04:03:58.727449 7703 raft_consensus.cc:2781] T 626d79b7b09940d4af54e73839f09a25 P 5fcb22667c684fa495415f7f391266c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:03:58.727475 7304 leader_election.cc:302] T 626d79b7b09940d4af54e73839f09a25 P 61a56ad7bdac46b199107e364bd82d4e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61a56ad7bdac46b199107e364bd82d4e, a46882de0a954f6b93c5449d6e09f340; no voters:
I20260808 04:03:58.727595 7703 raft_consensus.cc:482] T 626d79b7b09940d4af54e73839f09a25 P 5fcb22667c684fa495415f7f391266c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:03:58.727720 7703 raft_consensus.cc:3037] T 626d79b7b09940d4af54e73839f09a25 P 5fcb22667c684fa495415f7f391266c1 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:03:58.727911 7702 raft_consensus.cc:2781] T 626d79b7b09940d4af54e73839f09a25 P 61a56ad7bdac46b199107e364bd82d4e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:03:58.728073 7702 raft_consensus.cc:482] T 626d79b7b09940d4af54e73839f09a25 P 61a56ad7bdac46b199107e364bd82d4e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:03:58.728181 7702 raft_consensus.cc:3037] T 626d79b7b09940d4af54e73839f09a25 P 61a56ad7bdac46b199107e364bd82d4e [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:03:58.729990 7703 raft_consensus.cc:504] T 626d79b7b09940d4af54e73839f09a25 P 5fcb22667c684fa495415f7f391266c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46882de0a954f6b93c5449d6e09f340" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 39863 } } peers { permanent_uuid: "5fcb22667c684fa495415f7f391266c1" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42131 } } peers { permanent_uuid: "61a56ad7bdac46b199107e364bd82d4e" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 40595 } }
I20260808 04:03:58.730281 7702 raft_consensus.cc:504] T 626d79b7b09940d4af54e73839f09a25 P 61a56ad7bdac46b199107e364bd82d4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46882de0a954f6b93c5449d6e09f340" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 39863 } } peers { permanent_uuid: "5fcb22667c684fa495415f7f391266c1" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42131 } } peers { permanent_uuid: "61a56ad7bdac46b199107e364bd82d4e" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 40595 } }
I20260808 04:03:58.730592 7703 leader_election.cc:288] T 626d79b7b09940d4af54e73839f09a25 P 5fcb22667c684fa495415f7f391266c1 [CANDIDATE]: Term 1 election: Requested vote from peers a46882de0a954f6b93c5449d6e09f340 (127.5.83.3:39863), 61a56ad7bdac46b199107e364bd82d4e (127.5.83.2:40595)
I20260808 04:03:58.730909 7702 leader_election.cc:288] T 626d79b7b09940d4af54e73839f09a25 P 61a56ad7bdac46b199107e364bd82d4e [CANDIDATE]: Term 1 election: Requested vote from peers a46882de0a954f6b93c5449d6e09f340 (127.5.83.3:39863), 5fcb22667c684fa495415f7f391266c1 (127.5.83.1:42131)
I20260808 04:03:58.730947 7592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "626d79b7b09940d4af54e73839f09a25" candidate_uuid: "5fcb22667c684fa495415f7f391266c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a46882de0a954f6b93c5449d6e09f340"
I20260808 04:03:58.731130 7592 raft_consensus.cc:3037] T 626d79b7b09940d4af54e73839f09a25 P a46882de0a954f6b93c5449d6e09f340 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:03:58.731178 7400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "626d79b7b09940d4af54e73839f09a25" candidate_uuid: "5fcb22667c684fa495415f7f391266c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61a56ad7bdac46b199107e364bd82d4e"
I20260808 04:03:58.731561 7400 raft_consensus.cc:2379] T 626d79b7b09940d4af54e73839f09a25 P 61a56ad7bdac46b199107e364bd82d4e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 5fcb22667c684fa495415f7f391266c1 in current term 1: Already voted for candidate 61a56ad7bdac46b199107e364bd82d4e in this term.
I20260808 04:03:58.731629 7593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "626d79b7b09940d4af54e73839f09a25" candidate_uuid: "61a56ad7bdac46b199107e364bd82d4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a46882de0a954f6b93c5449d6e09f340"
I20260808 04:03:58.731639 7204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "626d79b7b09940d4af54e73839f09a25" candidate_uuid: "61a56ad7bdac46b199107e364bd82d4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcb22667c684fa495415f7f391266c1"
I20260808 04:03:58.731915 7204 raft_consensus.cc:2379] T 626d79b7b09940d4af54e73839f09a25 P 5fcb22667c684fa495415f7f391266c1 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 61a56ad7bdac46b199107e364bd82d4e in current term 1: Already voted for candidate 5fcb22667c684fa495415f7f391266c1 in this term.
I20260808 04:03:58.732566 7302 leader_election.cc:302] T 626d79b7b09940d4af54e73839f09a25 P 61a56ad7bdac46b199107e364bd82d4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61a56ad7bdac46b199107e364bd82d4e; no voters: 5fcb22667c684fa495415f7f391266c1, a46882de0a954f6b93c5449d6e09f340
I20260808 04:03:58.732860 7702 raft_consensus.cc:2726] T 626d79b7b09940d4af54e73839f09a25 P 61a56ad7bdac46b199107e364bd82d4e [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260808 04:03:58.733963 7592 raft_consensus.cc:2454] T 626d79b7b09940d4af54e73839f09a25 P a46882de0a954f6b93c5449d6e09f340 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5fcb22667c684fa495415f7f391266c1 in term 1.
I20260808 04:03:58.734266 7113 leader_election.cc:302] T 626d79b7b09940d4af54e73839f09a25 P 5fcb22667c684fa495415f7f391266c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5fcb22667c684fa495415f7f391266c1, a46882de0a954f6b93c5449d6e09f340; no voters: 61a56ad7bdac46b199107e364bd82d4e
I20260808 04:03:58.734516 7703 raft_consensus.cc:2781] T 626d79b7b09940d4af54e73839f09a25 P 5fcb22667c684fa495415f7f391266c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:03:58.734905 7703 raft_consensus.cc:686] T 626d79b7b09940d4af54e73839f09a25 P 5fcb22667c684fa495415f7f391266c1 [term 1 LEADER]: Becoming Leader. State: Replica: 5fcb22667c684fa495415f7f391266c1, State: Running, Role: LEADER
I20260808 04:03:58.735337 7703 consensus_queue.cc:237] T 626d79b7b09940d4af54e73839f09a25 P 5fcb22667c684fa495415f7f391266c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46882de0a954f6b93c5449d6e09f340" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 39863 } } peers { permanent_uuid: "5fcb22667c684fa495415f7f391266c1" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42131 } } peers { permanent_uuid: "61a56ad7bdac46b199107e364bd82d4e" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 40595 } }
I20260808 04:03:58.739665 6986 catalog_manager.cc:5495] T 626d79b7b09940d4af54e73839f09a25 P 5fcb22667c684fa495415f7f391266c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5fcb22667c684fa495415f7f391266c1 (127.5.83.1). New cstate: current_term: 1 leader_uuid: "5fcb22667c684fa495415f7f391266c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46882de0a954f6b93c5449d6e09f340" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 39863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5fcb22667c684fa495415f7f391266c1" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61a56ad7bdac46b199107e364bd82d4e" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 40595 } health_report { overall_health: UNKNOWN } } }
I20260808 04:03:58.773994 7691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:58.768365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39196 (local address 127.5.83.1:42131)
0808 04:03:58.768552 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:58.768559 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:58.768708 (+ 149us) server_negotiation.cc:407] Connection header received
0808 04:03:58.769043 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:58.769047 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:58.769144 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:58.769264 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:03:58.769633 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:58.770431 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:58.771789 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:58.772199 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:58.772310 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:58.772833 (+ 523us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:03:58.773374 (+ 541us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:03:58.773504 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:58.773633 (+ 129us) server_negotiation.cc:299] Negotiation successful
0808 04:03:58.773723 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260808 04:03:58.782703 7400 raft_consensus.cc:1261] T 626d79b7b09940d4af54e73839f09a25 P 61a56ad7bdac46b199107e364bd82d4e [term 1 FOLLOWER]: Refusing update from remote peer 5fcb22667c684fa495415f7f391266c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:03:58.783636 7592 raft_consensus.cc:1261] T 626d79b7b09940d4af54e73839f09a25 P a46882de0a954f6b93c5449d6e09f340 [term 1 FOLLOWER]: Refusing update from remote peer 5fcb22667c684fa495415f7f391266c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:03:58.783679 7715 consensus_queue.cc:1035] T 626d79b7b09940d4af54e73839f09a25 P 5fcb22667c684fa495415f7f391266c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61a56ad7bdac46b199107e364bd82d4e" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 40595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:03:58.784310 7703 consensus_queue.cc:1035] T 626d79b7b09940d4af54e73839f09a25 P 5fcb22667c684fa495415f7f391266c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a46882de0a954f6b93c5449d6e09f340" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:03:58.800117 7720 mvcc.cc:205] Tried to move back new op lower bound from 7316118891642056704 to 7316118891463327744. Current Snapshot: MvccSnapshot[applied={T|T < 7316118891642056704}]
I20260808 04:03:58.817510 6984 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44018:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:03:58.817893 6984 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44018:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:03:58.821664 6984 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0f18f42ae17b4626a802d0e17926c51a: Sending DeleteTablet for 3 replicas of tablet 626d79b7b09940d4af54e73839f09a25
I20260808 04:03:58.823101 7369 tablet_service.cc:1514] Processing DeleteTablet for tablet 626d79b7b09940d4af54e73839f09a25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:03:58 UTC) from {username='kudu', principal='kudu/127.5.83.62@KRBTEST.COM'} at 127.0.0.1:56342
I20260808 04:03:58.823091 7179 tablet_service.cc:1514] Processing DeleteTablet for tablet 626d79b7b09940d4af54e73839f09a25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:03:58 UTC) from {username='kudu', principal='kudu/127.5.83.62@KRBTEST.COM'} at 127.0.0.1:39184
I20260808 04:03:58.824548 7562 tablet_service.cc:1514] Processing DeleteTablet for tablet 626d79b7b09940d4af54e73839f09a25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:03:58 UTC) from {username='kudu', principal='kudu/127.5.83.62@KRBTEST.COM'} at 127.0.0.1:56462
I20260808 04:03:58.826452 6984 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44018:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:03:58.827191 6984 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:03:58.833285 7737 tablet_replica.cc:330] T 626d79b7b09940d4af54e73839f09a25 P a46882de0a954f6b93c5449d6e09f340: stopping tablet replica
I20260808 04:03:58.834143 7737 raft_consensus.cc:2229] T 626d79b7b09940d4af54e73839f09a25 P a46882de0a954f6b93c5449d6e09f340 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:03:58.834764 7737 raft_consensus.cc:2258] T 626d79b7b09940d4af54e73839f09a25 P a46882de0a954f6b93c5449d6e09f340 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:03:58.835999 7735 tablet_replica.cc:330] T 626d79b7b09940d4af54e73839f09a25 P 5fcb22667c684fa495415f7f391266c1: stopping tablet replica
I20260808 04:03:58.836999 7367 tablet_service.cc:1467] Processing CreateTablet for tablet 6553e5d9fcd041f480e89f8ef47d3ed0 (DEFAULT_TABLE table=test-table [id=cd5d25cb18dd40e0a6a0dbdde957be3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:03:58.837555 7367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6553e5d9fcd041f480e89f8ef47d3ed0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:03:58.839592 7737 ts_tablet_manager.cc:1905] T 626d79b7b09940d4af54e73839f09a25 P a46882de0a954f6b93c5449d6e09f340: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:03:58.840986 7735 raft_consensus.cc:2229] T 626d79b7b09940d4af54e73839f09a25 P 5fcb22667c684fa495415f7f391266c1 [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:03:58.842074 7179 tablet_service.cc:1467] Processing CreateTablet for tablet 6553e5d9fcd041f480e89f8ef47d3ed0 (DEFAULT_TABLE table=test-table [id=cd5d25cb18dd40e0a6a0dbdde957be3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:03:58.843801 7179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6553e5d9fcd041f480e89f8ef47d3ed0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:03:58.843976 7562 tablet_service.cc:1467] Processing CreateTablet for tablet 6553e5d9fcd041f480e89f8ef47d3ed0 (DEFAULT_TABLE table=test-table [id=cd5d25cb18dd40e0a6a0dbdde957be3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:03:58.842247 7735 raft_consensus.cc:2258] T 626d79b7b09940d4af54e73839f09a25 P 5fcb22667c684fa495415f7f391266c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:03:58.844436 7562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6553e5d9fcd041f480e89f8ef47d3ed0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:03:58.850797 7697 tablet_bootstrap.cc:492] T 6553e5d9fcd041f480e89f8ef47d3ed0 P 61a56ad7bdac46b199107e364bd82d4e: Bootstrap starting.
I20260808 04:03:58.851585 7735 ts_tablet_manager.cc:1905] T 626d79b7b09940d4af54e73839f09a25 P 5fcb22667c684fa495415f7f391266c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:03:58.852653 7697 tablet_bootstrap.cc:654] T 6553e5d9fcd041f480e89f8ef47d3ed0 P 61a56ad7bdac46b199107e364bd82d4e: Neither blocks nor log segments found. Creating new log.
I20260808 04:03:58.854259 7733 tablet_replica.cc:330] T 626d79b7b09940d4af54e73839f09a25 P 61a56ad7bdac46b199107e364bd82d4e: stopping tablet replica
I20260808 04:03:58.854800 7733 raft_consensus.cc:2229] T 626d79b7b09940d4af54e73839f09a25 P 61a56ad7bdac46b199107e364bd82d4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:03:58.855403 7733 raft_consensus.cc:2258] T 626d79b7b09940d4af54e73839f09a25 P 61a56ad7bdac46b199107e364bd82d4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:03:58.857985 7695 tablet_bootstrap.cc:492] T 6553e5d9fcd041f480e89f8ef47d3ed0 P 5fcb22667c684fa495415f7f391266c1: Bootstrap starting.
I20260808 04:03:58.859315 7733 ts_tablet_manager.cc:1905] T 626d79b7b09940d4af54e73839f09a25 P 61a56ad7bdac46b199107e364bd82d4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:03:58.859772 7695 tablet_bootstrap.cc:654] T 6553e5d9fcd041f480e89f8ef47d3ed0 P 5fcb22667c684fa495415f7f391266c1: Neither blocks nor log segments found. Creating new log.
I20260808 04:03:58.861974 7735 ts_tablet_manager.cc:1918] T 626d79b7b09940d4af54e73839f09a25 P 5fcb22667c684fa495415f7f391266c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:03:58.862278 7735 log.cc:1196] T 626d79b7b09940d4af54e73839f09a25 P 5fcb22667c684fa495415f7f391266c1: Deleting WAL directory at /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/minicluster-data/ts-0/wal/wals/626d79b7b09940d4af54e73839f09a25
I20260808 04:03:58.863020 7735 ts_tablet_manager.cc:1939] T 626d79b7b09940d4af54e73839f09a25 P 5fcb22667c684fa495415f7f391266c1: Deleting consensus metadata
I20260808 04:03:58.866097 7733 ts_tablet_manager.cc:1918] T 626d79b7b09940d4af54e73839f09a25 P 61a56ad7bdac46b199107e364bd82d4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:03:58.866269 7733 log.cc:1196] T 626d79b7b09940d4af54e73839f09a25 P 61a56ad7bdac46b199107e364bd82d4e: Deleting WAL directory at /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/minicluster-data/ts-1/wal/wals/626d79b7b09940d4af54e73839f09a25
I20260808 04:03:58.866099 6964 catalog_manager.cc:4841] TS 5fcb22667c684fa495415f7f391266c1 (127.5.83.1:42131): tablet 626d79b7b09940d4af54e73839f09a25 (table test-table [id=61904e4f42c44736965c5dd7b4ce0616]) successfully deleted
I20260808 04:03:58.866801 7733 ts_tablet_manager.cc:1939] T 626d79b7b09940d4af54e73839f09a25 P 61a56ad7bdac46b199107e364bd82d4e: Deleting consensus metadata
I20260808 04:03:58.867918 7694 tablet_bootstrap.cc:492] T 6553e5d9fcd041f480e89f8ef47d3ed0 P a46882de0a954f6b93c5449d6e09f340: Bootstrap starting.
I20260808 04:03:58.868494 6967 catalog_manager.cc:4841] TS 61a56ad7bdac46b199107e364bd82d4e (127.5.83.2:40595): tablet 626d79b7b09940d4af54e73839f09a25 (table test-table [id=61904e4f42c44736965c5dd7b4ce0616]) successfully deleted
I20260808 04:03:58.869644 7694 tablet_bootstrap.cc:654] T 6553e5d9fcd041f480e89f8ef47d3ed0 P a46882de0a954f6b93c5449d6e09f340: Neither blocks nor log segments found. Creating new log.
I20260808 04:03:58.870229 7695 tablet_bootstrap.cc:492] T 6553e5d9fcd041f480e89f8ef47d3ed0 P 5fcb22667c684fa495415f7f391266c1: No bootstrap required, opened a new log
I20260808 04:03:58.870378 7695 ts_tablet_manager.cc:1397] T 6553e5d9fcd041f480e89f8ef47d3ed0 P 5fcb22667c684fa495415f7f391266c1: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.005s
I20260808 04:03:58.871258 7695 raft_consensus.cc:348] T 6553e5d9fcd041f480e89f8ef47d3ed0 P 5fcb22667c684fa495415f7f391266c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fcb22667c684fa495415f7f391266c1" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42131 } } peers { permanent_uuid: "61a56ad7bdac46b199107e364bd82d4e" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 40595 } } peers { permanent_uuid: "a46882de0a954f6b93c5449d6e09f340" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 39863 } }
I20260808 04:03:58.871498 7695 raft_consensus.cc:374] T 6553e5d9fcd041f480e89f8ef47d3ed0 P 5fcb22667c684fa495415f7f391266c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:03:58.871619 7695 raft_consensus.cc:729] T 6553e5d9fcd041f480e89f8ef47d3ed0 P 5fcb22667c684fa495415f7f391266c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcb22667c684fa495415f7f391266c1, State: Initialized, Role: FOLLOWER
I20260808 04:03:58.871960 7695 consensus_queue.cc:260] T 6553e5d9fcd041f480e89f8ef47d3ed0 P 5fcb22667c684fa495415f7f391266c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fcb22667c684fa495415f7f391266c1" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42131 } } peers { permanent_uuid: "61a56ad7bdac46b199107e364bd82d4e" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 40595 } } peers { permanent_uuid: "a46882de0a954f6b93c5449d6e09f340" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 39863 } }
I20260808 04:03:58.872629 7737 ts_tablet_manager.cc:1918] T 626d79b7b09940d4af54e73839f09a25 P a46882de0a954f6b93c5449d6e09f340: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:03:58.872792 7695 ts_tablet_manager.cc:1428] T 6553e5d9fcd041f480e89f8ef47d3ed0 P 5fcb22667c684fa495415f7f391266c1: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260808 04:03:58.872812 7737 log.cc:1196] T 626d79b7b09940d4af54e73839f09a25 P a46882de0a954f6b93c5449d6e09f340: Deleting WAL directory at /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/minicluster-data/ts-2/wal/wals/626d79b7b09940d4af54e73839f09a25
I20260808 04:03:58.873308 7737 ts_tablet_manager.cc:1939] T 626d79b7b09940d4af54e73839f09a25 P a46882de0a954f6b93c5449d6e09f340: Deleting consensus metadata
I20260808 04:03:58.875214 6967 catalog_manager.cc:4841] TS a46882de0a954f6b93c5449d6e09f340 (127.5.83.3:39863): tablet 626d79b7b09940d4af54e73839f09a25 (table test-table [id=61904e4f42c44736965c5dd7b4ce0616]) successfully deleted
I20260808 04:03:58.876204 7694 tablet_bootstrap.cc:492] T 6553e5d9fcd041f480e89f8ef47d3ed0 P a46882de0a954f6b93c5449d6e09f340: No bootstrap required, opened a new log
I20260808 04:03:58.876400 7694 ts_tablet_manager.cc:1397] T 6553e5d9fcd041f480e89f8ef47d3ed0 P a46882de0a954f6b93c5449d6e09f340: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.006s
I20260808 04:03:58.876780 7697 tablet_bootstrap.cc:492] T 6553e5d9fcd041f480e89f8ef47d3ed0 P 61a56ad7bdac46b199107e364bd82d4e: No bootstrap required, opened a new log
I20260808 04:03:58.876938 7697 ts_tablet_manager.cc:1397] T 6553e5d9fcd041f480e89f8ef47d3ed0 P 61a56ad7bdac46b199107e364bd82d4e: Time spent bootstrapping tablet: real 0.026s user 0.000s sys 0.011s
I20260808 04:03:58.877302 7694 raft_consensus.cc:348] T 6553e5d9fcd041f480e89f8ef47d3ed0 P a46882de0a954f6b93c5449d6e09f340 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fcb22667c684fa495415f7f391266c1" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42131 } } peers { permanent_uuid: "61a56ad7bdac46b199107e364bd82d4e" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 40595 } } peers { permanent_uuid: "a46882de0a954f6b93c5449d6e09f340" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 39863 } }
I20260808 04:03:58.877472 7694 raft_consensus.cc:374] T 6553e5d9fcd041f480e89f8ef47d3ed0 P a46882de0a954f6b93c5449d6e09f340 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:03:58.877564 7694 raft_consensus.cc:729] T 6553e5d9fcd041f480e89f8ef47d3ed0 P a46882de0a954f6b93c5449d6e09f340 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a46882de0a954f6b93c5449d6e09f340, State: Initialized, Role: FOLLOWER
I20260808 04:03:58.877712 7697 raft_consensus.cc:348] T 6553e5d9fcd041f480e89f8ef47d3ed0 P 61a56ad7bdac46b199107e364bd82d4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fcb22667c684fa495415f7f391266c1" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42131 } } peers { permanent_uuid: "61a56ad7bdac46b199107e364bd82d4e" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 40595 } } peers { permanent_uuid: "a46882de0a954f6b93c5449d6e09f340" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 39863 } }
I20260808 04:03:58.877800 7694 consensus_queue.cc:260] T 6553e5d9fcd041f480e89f8ef47d3ed0 P a46882de0a954f6b93c5449d6e09f340 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fcb22667c684fa495415f7f391266c1" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42131 } } peers { permanent_uuid: "61a56ad7bdac46b199107e364bd82d4e" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 40595 } } peers { permanent_uuid: "a46882de0a954f6b93c5449d6e09f340" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 39863 } }
I20260808 04:03:58.877933 7697 raft_consensus.cc:374] T 6553e5d9fcd041f480e89f8ef47d3ed0 P 61a56ad7bdac46b199107e364bd82d4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:03:58.878032 7697 raft_consensus.cc:729] T 6553e5d9fcd041f480e89f8ef47d3ed0 P 61a56ad7bdac46b199107e364bd82d4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61a56ad7bdac46b199107e364bd82d4e, State: Initialized, Role: FOLLOWER
I20260808 04:03:58.878360 7694 ts_tablet_manager.cc:1428] T 6553e5d9fcd041f480e89f8ef47d3ed0 P a46882de0a954f6b93c5449d6e09f340: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260808 04:03:58.878264 7697 consensus_queue.cc:260] T 6553e5d9fcd041f480e89f8ef47d3ed0 P 61a56ad7bdac46b199107e364bd82d4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fcb22667c684fa495415f7f391266c1" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42131 } } peers { permanent_uuid: "61a56ad7bdac46b199107e364bd82d4e" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 40595 } } peers { permanent_uuid: "a46882de0a954f6b93c5449d6e09f340" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 39863 } }
I20260808 04:03:58.878837 7697 ts_tablet_manager.cc:1428] T 6553e5d9fcd041f480e89f8ef47d3ed0 P 61a56ad7bdac46b199107e364bd82d4e: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260808 04:03:59.036648 7728 raft_consensus.cc:482] T 6553e5d9fcd041f480e89f8ef47d3ed0 P 5fcb22667c684fa495415f7f391266c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:03:59.036942 7728 raft_consensus.cc:504] T 6553e5d9fcd041f480e89f8ef47d3ed0 P 5fcb22667c684fa495415f7f391266c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fcb22667c684fa495415f7f391266c1" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42131 } } peers { permanent_uuid: "61a56ad7bdac46b199107e364bd82d4e" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 40595 } } peers { permanent_uuid: "a46882de0a954f6b93c5449d6e09f340" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 39863 } }
I20260808 04:03:59.037736 7728 leader_election.cc:288] T 6553e5d9fcd041f480e89f8ef47d3ed0 P 5fcb22667c684fa495415f7f391266c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61a56ad7bdac46b199107e364bd82d4e (127.5.83.2:40595), a46882de0a954f6b93c5449d6e09f340 (127.5.83.3:39863)
I20260808 04:03:59.038332 7400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6553e5d9fcd041f480e89f8ef47d3ed0" candidate_uuid: "5fcb22667c684fa495415f7f391266c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61a56ad7bdac46b199107e364bd82d4e" is_pre_election: true
I20260808 04:03:59.038414 7592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6553e5d9fcd041f480e89f8ef47d3ed0" candidate_uuid: "5fcb22667c684fa495415f7f391266c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a46882de0a954f6b93c5449d6e09f340" is_pre_election: true
I20260808 04:03:59.038589 7400 raft_consensus.cc:2454] T 6553e5d9fcd041f480e89f8ef47d3ed0 P 61a56ad7bdac46b199107e364bd82d4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5fcb22667c684fa495415f7f391266c1 in term 0.
I20260808 04:03:59.038714 7592 raft_consensus.cc:2454] T 6553e5d9fcd041f480e89f8ef47d3ed0 P a46882de0a954f6b93c5449d6e09f340 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5fcb22667c684fa495415f7f391266c1 in term 0.
I20260808 04:03:59.039070 7113 leader_election.cc:302] T 6553e5d9fcd041f480e89f8ef47d3ed0 P 5fcb22667c684fa495415f7f391266c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5fcb22667c684fa495415f7f391266c1, 61a56ad7bdac46b199107e364bd82d4e; no voters:
I20260808 04:03:59.039352 7728 raft_consensus.cc:2781] T 6553e5d9fcd041f480e89f8ef47d3ed0 P 5fcb22667c684fa495415f7f391266c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:03:59.039539 7728 raft_consensus.cc:482] T 6553e5d9fcd041f480e89f8ef47d3ed0 P 5fcb22667c684fa495415f7f391266c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:03:59.039644 7728 raft_consensus.cc:3037] T 6553e5d9fcd041f480e89f8ef47d3ed0 P 5fcb22667c684fa495415f7f391266c1 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:03:59.041935 7728 raft_consensus.cc:504] T 6553e5d9fcd041f480e89f8ef47d3ed0 P 5fcb22667c684fa495415f7f391266c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fcb22667c684fa495415f7f391266c1" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42131 } } peers { permanent_uuid: "61a56ad7bdac46b199107e364bd82d4e" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 40595 } } peers { permanent_uuid: "a46882de0a954f6b93c5449d6e09f340" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 39863 } }
I20260808 04:03:59.042461 7728 leader_election.cc:288] T 6553e5d9fcd041f480e89f8ef47d3ed0 P 5fcb22667c684fa495415f7f391266c1 [CANDIDATE]: Term 1 election: Requested vote from peers 61a56ad7bdac46b199107e364bd82d4e (127.5.83.2:40595), a46882de0a954f6b93c5449d6e09f340 (127.5.83.3:39863)
I20260808 04:03:59.042919 7400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6553e5d9fcd041f480e89f8ef47d3ed0" candidate_uuid: "5fcb22667c684fa495415f7f391266c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61a56ad7bdac46b199107e364bd82d4e"
I20260808 04:03:59.043011 7592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6553e5d9fcd041f480e89f8ef47d3ed0" candidate_uuid: "5fcb22667c684fa495415f7f391266c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a46882de0a954f6b93c5449d6e09f340"
I20260808 04:03:59.043115 7400 raft_consensus.cc:3037] T 6553e5d9fcd041f480e89f8ef47d3ed0 P 61a56ad7bdac46b199107e364bd82d4e [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:03:59.043208 7592 raft_consensus.cc:3037] T 6553e5d9fcd041f480e89f8ef47d3ed0 P a46882de0a954f6b93c5449d6e09f340 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:03:59.045187 7400 raft_consensus.cc:2454] T 6553e5d9fcd041f480e89f8ef47d3ed0 P 61a56ad7bdac46b199107e364bd82d4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5fcb22667c684fa495415f7f391266c1 in term 1.
I20260808 04:03:59.045578 7113 leader_election.cc:302] T 6553e5d9fcd041f480e89f8ef47d3ed0 P 5fcb22667c684fa495415f7f391266c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5fcb22667c684fa495415f7f391266c1, 61a56ad7bdac46b199107e364bd82d4e; no voters:
I20260808 04:03:59.045836 7728 raft_consensus.cc:2781] T 6553e5d9fcd041f480e89f8ef47d3ed0 P 5fcb22667c684fa495415f7f391266c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:03:59.046069 7592 raft_consensus.cc:2454] T 6553e5d9fcd041f480e89f8ef47d3ed0 P a46882de0a954f6b93c5449d6e09f340 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5fcb22667c684fa495415f7f391266c1 in term 1.
I20260808 04:03:59.045992 7728 raft_consensus.cc:686] T 6553e5d9fcd041f480e89f8ef47d3ed0 P 5fcb22667c684fa495415f7f391266c1 [term 1 LEADER]: Becoming Leader. State: Replica: 5fcb22667c684fa495415f7f391266c1, State: Running, Role: LEADER
I20260808 04:03:59.046543 7728 consensus_queue.cc:237] T 6553e5d9fcd041f480e89f8ef47d3ed0 P 5fcb22667c684fa495415f7f391266c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fcb22667c684fa495415f7f391266c1" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42131 } } peers { permanent_uuid: "61a56ad7bdac46b199107e364bd82d4e" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 40595 } } peers { permanent_uuid: "a46882de0a954f6b93c5449d6e09f340" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 39863 } }
I20260808 04:03:59.049263 6985 catalog_manager.cc:5495] T 6553e5d9fcd041f480e89f8ef47d3ed0 P 5fcb22667c684fa495415f7f391266c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5fcb22667c684fa495415f7f391266c1 (127.5.83.1). New cstate: current_term: 1 leader_uuid: "5fcb22667c684fa495415f7f391266c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fcb22667c684fa495415f7f391266c1" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61a56ad7bdac46b199107e364bd82d4e" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 40595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a46882de0a954f6b93c5449d6e09f340" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 39863 } health_report { overall_health: UNKNOWN } } }
Aug 08 04:03:59 dist-test-slave-1l3l krb5kdc[6923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161836, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.62@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:03:59.116698 7753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:59.101866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:45699 (local address 127.0.0.1:44028)
0808 04:03:59.102670 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:59.102689 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 04:03:59.102830 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:03:59.103131 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:03:59.103135 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:03:59.103915 (+ 780us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:03:59.104132 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:59.104142 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:59.105184 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:59.105189 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:59.106135 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:59.106146 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:59.106834 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:59.106840 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:59.107016 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:59.107740 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:03:59.107757 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:03:59.110218 (+ 2461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:03:59.114130 (+ 3912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:03:59.114140 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:03:59.114145 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:03:59.114544 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:03:59.114962 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:03:59.114966 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:03:59.114968 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:03:59.115146 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:03:59.115816 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:03:59.115822 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:03:59.115994 (+ 172us) client_negotiation.cc:769] Sending connection context
0808 04:03:59.116165 (+ 171us) client_negotiation.cc:240] Negotiation successful
0808 04:03:59.116348 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":697,"thread_start_us":569,"threads_started":1}
I20260808 04:03:59.117229 7751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:59.101901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44028 (local address 127.5.83.62:45699)
0808 04:03:59.102503 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:59.102510 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:59.102773 (+ 263us) server_negotiation.cc:407] Connection header received
0808 04:03:59.102942 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:59.102946 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:59.103006 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:59.103091 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:59.104291 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:59.105059 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:59.106287 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:59.106683 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:59.106786 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:59.110446 (+ 3660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:59.110454 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:59.110459 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:03:59.110498 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:59.113992 (+ 3494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:59.114670 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:59.114675 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:59.114678 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:59.114850 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:59.115258 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:59.115263 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:59.115267 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:59.115660 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:59.115841 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:59.116299 (+ 458us) server_negotiation.cc:299] Negotiation successful
0808 04:03:59.116548 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":300,"threads_started":1}
I20260808 04:03:59.122208 6985 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.5.83.62@KRBTEST.COM'} at 127.0.0.1:44028:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260808 04:03:59.123378 6985 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:03:59.130380 7179 tablet_service.cc:1467] Processing CreateTablet for tablet 323f5cf487fd417c940f794fed443f92 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bf73be761cb44bb0aa2fc93bc2528ffd]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 04:03:59.130518 7367 tablet_service.cc:1467] Processing CreateTablet for tablet 323f5cf487fd417c940f794fed443f92 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bf73be761cb44bb0aa2fc93bc2528ffd]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 04:03:59.130975 7179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 323f5cf487fd417c940f794fed443f92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:03:59.131023 7367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 323f5cf487fd417c940f794fed443f92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:03:59.133438 7562 tablet_service.cc:1467] Processing CreateTablet for tablet 323f5cf487fd417c940f794fed443f92 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bf73be761cb44bb0aa2fc93bc2528ffd]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 04:03:59.134033 7562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 323f5cf487fd417c940f794fed443f92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:03:59.137049 7697 tablet_bootstrap.cc:492] T 323f5cf487fd417c940f794fed443f92 P 61a56ad7bdac46b199107e364bd82d4e: Bootstrap starting.
I20260808 04:03:59.138513 7695 tablet_bootstrap.cc:492] T 323f5cf487fd417c940f794fed443f92 P 5fcb22667c684fa495415f7f391266c1: Bootstrap starting.
I20260808 04:03:59.138640 7697 tablet_bootstrap.cc:654] T 323f5cf487fd417c940f794fed443f92 P 61a56ad7bdac46b199107e364bd82d4e: Neither blocks nor log segments found. Creating new log.
I20260808 04:03:59.140198 7695 tablet_bootstrap.cc:654] T 323f5cf487fd417c940f794fed443f92 P 5fcb22667c684fa495415f7f391266c1: Neither blocks nor log segments found. Creating new log.
I20260808 04:03:59.140275 7694 tablet_bootstrap.cc:492] T 323f5cf487fd417c940f794fed443f92 P a46882de0a954f6b93c5449d6e09f340: Bootstrap starting.
I20260808 04:03:59.141873 7694 tablet_bootstrap.cc:654] T 323f5cf487fd417c940f794fed443f92 P a46882de0a954f6b93c5449d6e09f340: Neither blocks nor log segments found. Creating new log.
I20260808 04:03:59.150830 7697 tablet_bootstrap.cc:492] T 323f5cf487fd417c940f794fed443f92 P 61a56ad7bdac46b199107e364bd82d4e: No bootstrap required, opened a new log
I20260808 04:03:59.151036 7697 ts_tablet_manager.cc:1397] T 323f5cf487fd417c940f794fed443f92 P 61a56ad7bdac46b199107e364bd82d4e: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.004s
I20260808 04:03:59.151518 7694 tablet_bootstrap.cc:492] T 323f5cf487fd417c940f794fed443f92 P a46882de0a954f6b93c5449d6e09f340: No bootstrap required, opened a new log
I20260808 04:03:59.151688 7694 ts_tablet_manager.cc:1397] T 323f5cf487fd417c940f794fed443f92 P a46882de0a954f6b93c5449d6e09f340: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.009s
I20260808 04:03:59.151784 7697 raft_consensus.cc:348] T 323f5cf487fd417c940f794fed443f92 P 61a56ad7bdac46b199107e364bd82d4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fcb22667c684fa495415f7f391266c1" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42131 } } peers { permanent_uuid: "61a56ad7bdac46b199107e364bd82d4e" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 40595 } } peers { permanent_uuid: "a46882de0a954f6b93c5449d6e09f340" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 39863 } }
I20260808 04:03:59.152078 7697 raft_consensus.cc:374] T 323f5cf487fd417c940f794fed443f92 P 61a56ad7bdac46b199107e364bd82d4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:03:59.152169 7697 raft_consensus.cc:729] T 323f5cf487fd417c940f794fed443f92 P 61a56ad7bdac46b199107e364bd82d4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61a56ad7bdac46b199107e364bd82d4e, State: Initialized, Role: FOLLOWER
I20260808 04:03:59.152480 7697 consensus_queue.cc:260] T 323f5cf487fd417c940f794fed443f92 P 61a56ad7bdac46b199107e364bd82d4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fcb22667c684fa495415f7f391266c1" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42131 } } peers { permanent_uuid: "61a56ad7bdac46b199107e364bd82d4e" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 40595 } } peers { permanent_uuid: "a46882de0a954f6b93c5449d6e09f340" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 39863 } }
I20260808 04:03:59.152539 7694 raft_consensus.cc:348] T 323f5cf487fd417c940f794fed443f92 P a46882de0a954f6b93c5449d6e09f340 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fcb22667c684fa495415f7f391266c1" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42131 } } peers { permanent_uuid: "61a56ad7bdac46b199107e364bd82d4e" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 40595 } } peers { permanent_uuid: "a46882de0a954f6b93c5449d6e09f340" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 39863 } }
I20260808 04:03:59.152786 7694 raft_consensus.cc:374] T 323f5cf487fd417c940f794fed443f92 P a46882de0a954f6b93c5449d6e09f340 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:03:59.152890 7694 raft_consensus.cc:729] T 323f5cf487fd417c940f794fed443f92 P a46882de0a954f6b93c5449d6e09f340 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a46882de0a954f6b93c5449d6e09f340, State: Initialized, Role: FOLLOWER
I20260808 04:03:59.152992 7702 tablet_replica.cc:439] T 323f5cf487fd417c940f794fed443f92 P 61a56ad7bdac46b199107e364bd82d4e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fcb22667c684fa495415f7f391266c1" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42131 } } peers { permanent_uuid: "61a56ad7bdac46b199107e364bd82d4e" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 40595 } } peers { permanent_uuid: "a46882de0a954f6b93c5449d6e09f340" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 39863 } } }
I20260808 04:03:59.153276 7702 tablet_replica.cc:442] T 323f5cf487fd417c940f794fed443f92 P 61a56ad7bdac46b199107e364bd82d4e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 04:03:59.153156 7694 consensus_queue.cc:260] T 323f5cf487fd417c940f794fed443f92 P a46882de0a954f6b93c5449d6e09f340 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fcb22667c684fa495415f7f391266c1" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42131 } } peers { permanent_uuid: "61a56ad7bdac46b199107e364bd82d4e" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 40595 } } peers { permanent_uuid: "a46882de0a954f6b93c5449d6e09f340" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 39863 } }
I20260808 04:03:59.153581 7704 tablet_replica.cc:439] T 323f5cf487fd417c940f794fed443f92 P a46882de0a954f6b93c5449d6e09f340: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fcb22667c684fa495415f7f391266c1" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42131 } } peers { permanent_uuid: "61a56ad7bdac46b199107e364bd82d4e" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 40595 } } peers { permanent_uuid: "a46882de0a954f6b93c5449d6e09f340" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 39863 } } }
I20260808 04:03:59.153891 7704 tablet_replica.cc:442] T 323f5cf487fd417c940f794fed443f92 P a46882de0a954f6b93c5449d6e09f340: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 04:03:59.154732 7695 tablet_bootstrap.cc:492] T 323f5cf487fd417c940f794fed443f92 P 5fcb22667c684fa495415f7f391266c1: No bootstrap required, opened a new log
I20260808 04:03:59.154978 7695 ts_tablet_manager.cc:1397] T 323f5cf487fd417c940f794fed443f92 P 5fcb22667c684fa495415f7f391266c1: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.001s
I20260808 04:03:59.155614 7697 ts_tablet_manager.cc:1428] T 323f5cf487fd417c940f794fed443f92 P 61a56ad7bdac46b199107e364bd82d4e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260808 04:03:59.155954 7695 raft_consensus.cc:348] T 323f5cf487fd417c940f794fed443f92 P 5fcb22667c684fa495415f7f391266c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fcb22667c684fa495415f7f391266c1" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42131 } } peers { permanent_uuid: "61a56ad7bdac46b199107e364bd82d4e" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 40595 } } peers { permanent_uuid: "a46882de0a954f6b93c5449d6e09f340" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 39863 } }
I20260808 04:03:59.156314 7695 raft_consensus.cc:374] T 323f5cf487fd417c940f794fed443f92 P 5fcb22667c684fa495415f7f391266c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:03:59.156468 7695 raft_consensus.cc:729] T 323f5cf487fd417c940f794fed443f92 P 5fcb22667c684fa495415f7f391266c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcb22667c684fa495415f7f391266c1, State: Initialized, Role: FOLLOWER
I20260808 04:03:59.156774 7695 consensus_queue.cc:260] T 323f5cf487fd417c940f794fed443f92 P 5fcb22667c684fa495415f7f391266c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fcb22667c684fa495415f7f391266c1" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42131 } } peers { permanent_uuid: "61a56ad7bdac46b199107e364bd82d4e" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 40595 } } peers { permanent_uuid: "a46882de0a954f6b93c5449d6e09f340" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 39863 } }
I20260808 04:03:59.157178 7694 ts_tablet_manager.cc:1428] T 323f5cf487fd417c940f794fed443f92 P a46882de0a954f6b93c5449d6e09f340: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20260808 04:03:59.157395 7728 tablet_replica.cc:439] T 323f5cf487fd417c940f794fed443f92 P 5fcb22667c684fa495415f7f391266c1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fcb22667c684fa495415f7f391266c1" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42131 } } peers { permanent_uuid: "61a56ad7bdac46b199107e364bd82d4e" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 40595 } } peers { permanent_uuid: "a46882de0a954f6b93c5449d6e09f340" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 39863 } } }
I20260808 04:03:59.157719 7728 tablet_replica.cc:442] T 323f5cf487fd417c940f794fed443f92 P 5fcb22667c684fa495415f7f391266c1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 04:03:59.158545 7695 ts_tablet_manager.cc:1428] T 323f5cf487fd417c940f794fed443f92 P 5fcb22667c684fa495415f7f391266c1: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260808 04:03:59.284897 7702 raft_consensus.cc:482] T 323f5cf487fd417c940f794fed443f92 P 61a56ad7bdac46b199107e364bd82d4e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:03:59.285076 7702 raft_consensus.cc:504] T 323f5cf487fd417c940f794fed443f92 P 61a56ad7bdac46b199107e364bd82d4e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fcb22667c684fa495415f7f391266c1" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42131 } } peers { permanent_uuid: "61a56ad7bdac46b199107e364bd82d4e" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 40595 } } peers { permanent_uuid: "a46882de0a954f6b93c5449d6e09f340" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 39863 } }
I20260808 04:03:59.285622 7702 leader_election.cc:288] T 323f5cf487fd417c940f794fed443f92 P 61a56ad7bdac46b199107e364bd82d4e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5fcb22667c684fa495415f7f391266c1 (127.5.83.1:42131), a46882de0a954f6b93c5449d6e09f340 (127.5.83.3:39863)
I20260808 04:03:59.286180 7592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "323f5cf487fd417c940f794fed443f92" candidate_uuid: "61a56ad7bdac46b199107e364bd82d4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a46882de0a954f6b93c5449d6e09f340" is_pre_election: true
I20260808 04:03:59.286196 7204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "323f5cf487fd417c940f794fed443f92" candidate_uuid: "61a56ad7bdac46b199107e364bd82d4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcb22667c684fa495415f7f391266c1" is_pre_election: true
I20260808 04:03:59.286419 7592 raft_consensus.cc:2454] T 323f5cf487fd417c940f794fed443f92 P a46882de0a954f6b93c5449d6e09f340 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61a56ad7bdac46b199107e364bd82d4e in term 0.
I20260808 04:03:59.286444 7204 raft_consensus.cc:2454] T 323f5cf487fd417c940f794fed443f92 P 5fcb22667c684fa495415f7f391266c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61a56ad7bdac46b199107e364bd82d4e in term 0.
I20260808 04:03:59.286815 7304 leader_election.cc:302] T 323f5cf487fd417c940f794fed443f92 P 61a56ad7bdac46b199107e364bd82d4e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61a56ad7bdac46b199107e364bd82d4e, a46882de0a954f6b93c5449d6e09f340; no voters:
I20260808 04:03:59.287101 7702 raft_consensus.cc:2781] T 323f5cf487fd417c940f794fed443f92 P 61a56ad7bdac46b199107e364bd82d4e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:03:59.287200 7702 raft_consensus.cc:482] T 323f5cf487fd417c940f794fed443f92 P 61a56ad7bdac46b199107e364bd82d4e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:03:59.287307 7702 raft_consensus.cc:3037] T 323f5cf487fd417c940f794fed443f92 P 61a56ad7bdac46b199107e364bd82d4e [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:03:59.289351 7702 raft_consensus.cc:504] T 323f5cf487fd417c940f794fed443f92 P 61a56ad7bdac46b199107e364bd82d4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fcb22667c684fa495415f7f391266c1" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42131 } } peers { permanent_uuid: "61a56ad7bdac46b199107e364bd82d4e" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 40595 } } peers { permanent_uuid: "a46882de0a954f6b93c5449d6e09f340" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 39863 } }
I20260808 04:03:59.289868 7702 leader_election.cc:288] T 323f5cf487fd417c940f794fed443f92 P 61a56ad7bdac46b199107e364bd82d4e [CANDIDATE]: Term 1 election: Requested vote from peers 5fcb22667c684fa495415f7f391266c1 (127.5.83.1:42131), a46882de0a954f6b93c5449d6e09f340 (127.5.83.3:39863)
I20260808 04:03:59.290253 7204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "323f5cf487fd417c940f794fed443f92" candidate_uuid: "61a56ad7bdac46b199107e364bd82d4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcb22667c684fa495415f7f391266c1"
I20260808 04:03:59.290309 7592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "323f5cf487fd417c940f794fed443f92" candidate_uuid: "61a56ad7bdac46b199107e364bd82d4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a46882de0a954f6b93c5449d6e09f340"
I20260808 04:03:59.290427 7204 raft_consensus.cc:3037] T 323f5cf487fd417c940f794fed443f92 P 5fcb22667c684fa495415f7f391266c1 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:03:59.290516 7592 raft_consensus.cc:3037] T 323f5cf487fd417c940f794fed443f92 P a46882de0a954f6b93c5449d6e09f340 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:03:59.292544 7592 raft_consensus.cc:2454] T 323f5cf487fd417c940f794fed443f92 P a46882de0a954f6b93c5449d6e09f340 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61a56ad7bdac46b199107e364bd82d4e in term 1.
I20260808 04:03:59.292620 7204 raft_consensus.cc:2454] T 323f5cf487fd417c940f794fed443f92 P 5fcb22667c684fa495415f7f391266c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61a56ad7bdac46b199107e364bd82d4e in term 1.
I20260808 04:03:59.292944 7304 leader_election.cc:302] T 323f5cf487fd417c940f794fed443f92 P 61a56ad7bdac46b199107e364bd82d4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61a56ad7bdac46b199107e364bd82d4e, a46882de0a954f6b93c5449d6e09f340; no voters:
I20260808 04:03:59.293205 7702 raft_consensus.cc:2781] T 323f5cf487fd417c940f794fed443f92 P 61a56ad7bdac46b199107e364bd82d4e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:03:59.293601 7702 raft_consensus.cc:686] T 323f5cf487fd417c940f794fed443f92 P 61a56ad7bdac46b199107e364bd82d4e [term 1 LEADER]: Becoming Leader. State: Replica: 61a56ad7bdac46b199107e364bd82d4e, State: Running, Role: LEADER
I20260808 04:03:59.293982 7702 consensus_queue.cc:237] T 323f5cf487fd417c940f794fed443f92 P 61a56ad7bdac46b199107e364bd82d4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fcb22667c684fa495415f7f391266c1" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42131 } } peers { permanent_uuid: "61a56ad7bdac46b199107e364bd82d4e" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 40595 } } peers { permanent_uuid: "a46882de0a954f6b93c5449d6e09f340" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 39863 } }
I20260808 04:03:59.296033 7759 tablet_replica.cc:439] T 323f5cf487fd417c940f794fed443f92 P 61a56ad7bdac46b199107e364bd82d4e: TxnStatusTablet state changed. Reason: New leader 61a56ad7bdac46b199107e364bd82d4e. Latest consensus state: current_term: 1 leader_uuid: "61a56ad7bdac46b199107e364bd82d4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fcb22667c684fa495415f7f391266c1" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42131 } } peers { permanent_uuid: "61a56ad7bdac46b199107e364bd82d4e" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 40595 } } peers { permanent_uuid: "a46882de0a954f6b93c5449d6e09f340" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 39863 } } }
I20260808 04:03:59.296236 7759 tablet_replica.cc:442] T 323f5cf487fd417c940f794fed443f92 P 61a56ad7bdac46b199107e364bd82d4e: This TxnStatusTablet replica's current role is: LEADER
I20260808 04:03:59.296833 7762 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260808 04:03:59.297108 7762 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260808 04:03:59.297427 6985 catalog_manager.cc:5495] T 323f5cf487fd417c940f794fed443f92 P 61a56ad7bdac46b199107e364bd82d4e reported cstate change: term changed from 0 to 1, leader changed from <none> to 61a56ad7bdac46b199107e364bd82d4e (127.5.83.2). New cstate: current_term: 1 leader_uuid: "61a56ad7bdac46b199107e364bd82d4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fcb22667c684fa495415f7f391266c1" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61a56ad7bdac46b199107e364bd82d4e" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 40595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a46882de0a954f6b93c5449d6e09f340" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 39863 } health_report { overall_health: UNKNOWN } } }
I20260808 04:03:59.334712 7753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:59.328799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.2:40595 (local address 127.0.0.1:56350)
0808 04:03:59.329049 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:59.329071 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 04:03:59.329211 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:03:59.329816 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:03:59.329821 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:03:59.329833 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 04:03:59.330021 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:59.330031 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:59.331177 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:59.331181 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:59.332258 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:59.332270 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:59.333047 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:59.333053 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:59.333224 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:59.333292 (+ 68us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 04:03:59.334123 (+ 831us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 04:03:59.334145 (+ 22us) client_negotiation.cc:769] Sending connection context
0808 04:03:59.334224 (+ 79us) client_negotiation.cc:240] Negotiation successful
0808 04:03:59.334372 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260808 04:03:59.334738 7768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:59.328813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56350 (local address 127.5.83.2:40595)
0808 04:03:59.329480 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:59.329485 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:59.329498 (+ 13us) server_negotiation.cc:407] Connection header received
0808 04:03:59.329569 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:59.329573 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:59.329641 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:59.329771 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:03:59.330181 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:59.331035 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:59.332408 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:59.332900 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:59.333009 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:59.333488 (+ 479us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:03:59.334019 (+ 531us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:03:59.334148 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:59.334269 (+ 121us) server_negotiation.cc:299] Negotiation successful
0808 04:03:59.334347 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":340,"threads_started":1}
I20260808 04:03:59.339560 7592 raft_consensus.cc:1261] T 323f5cf487fd417c940f794fed443f92 P a46882de0a954f6b93c5449d6e09f340 [term 1 FOLLOWER]: Refusing update from remote peer 61a56ad7bdac46b199107e364bd82d4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:03:59.339772 7204 raft_consensus.cc:1261] T 323f5cf487fd417c940f794fed443f92 P 5fcb22667c684fa495415f7f391266c1 [term 1 FOLLOWER]: Refusing update from remote peer 61a56ad7bdac46b199107e364bd82d4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:03:59.340283 7759 consensus_queue.cc:1035] T 323f5cf487fd417c940f794fed443f92 P 61a56ad7bdac46b199107e364bd82d4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a46882de0a954f6b93c5449d6e09f340" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:03:59.340641 7702 consensus_queue.cc:1035] T 323f5cf487fd417c940f794fed443f92 P 61a56ad7bdac46b199107e364bd82d4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fcb22667c684fa495415f7f391266c1" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:03:59.349346 7704 tablet_replica.cc:439] T 323f5cf487fd417c940f794fed443f92 P a46882de0a954f6b93c5449d6e09f340: TxnStatusTablet state changed. Reason: New leader 61a56ad7bdac46b199107e364bd82d4e. Latest consensus state: current_term: 1 leader_uuid: "61a56ad7bdac46b199107e364bd82d4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fcb22667c684fa495415f7f391266c1" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42131 } } peers { permanent_uuid: "61a56ad7bdac46b199107e364bd82d4e" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 40595 } } peers { permanent_uuid: "a46882de0a954f6b93c5449d6e09f340" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 39863 } } }
I20260808 04:03:59.349615 7704 tablet_replica.cc:442] T 323f5cf487fd417c940f794fed443f92 P a46882de0a954f6b93c5449d6e09f340: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 04:03:59.350948 7769 mvcc.cc:205] Tried to move back new op lower bound from 7316118893927936000 to 7316118893751271424. Current Snapshot: MvccSnapshot[applied={T|T < 7316118893927936000}]
I20260808 04:03:59.357980 7704 tablet_replica.cc:439] T 323f5cf487fd417c940f794fed443f92 P a46882de0a954f6b93c5449d6e09f340: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "61a56ad7bdac46b199107e364bd82d4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fcb22667c684fa495415f7f391266c1" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42131 } } peers { permanent_uuid: "61a56ad7bdac46b199107e364bd82d4e" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 40595 } } peers { permanent_uuid: "a46882de0a954f6b93c5449d6e09f340" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 39863 } } }
I20260808 04:03:59.358212 7704 tablet_replica.cc:442] T 323f5cf487fd417c940f794fed443f92 P a46882de0a954f6b93c5449d6e09f340: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 04:03:59.357224 7702 tablet_replica.cc:439] T 323f5cf487fd417c940f794fed443f92 P 61a56ad7bdac46b199107e364bd82d4e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "61a56ad7bdac46b199107e364bd82d4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fcb22667c684fa495415f7f391266c1" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42131 } } peers { permanent_uuid: "61a56ad7bdac46b199107e364bd82d4e" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 40595 } } peers { permanent_uuid: "a46882de0a954f6b93c5449d6e09f340" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 39863 } } }
I20260808 04:03:59.360383 7702 tablet_replica.cc:442] T 323f5cf487fd417c940f794fed443f92 P 61a56ad7bdac46b199107e364bd82d4e: This TxnStatusTablet replica's current role is: LEADER
I20260808 04:03:59.364548 7728 tablet_replica.cc:439] T 323f5cf487fd417c940f794fed443f92 P 5fcb22667c684fa495415f7f391266c1: TxnStatusTablet state changed. Reason: New leader 61a56ad7bdac46b199107e364bd82d4e. Latest consensus state: current_term: 1 leader_uuid: "61a56ad7bdac46b199107e364bd82d4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fcb22667c684fa495415f7f391266c1" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42131 } } peers { permanent_uuid: "61a56ad7bdac46b199107e364bd82d4e" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 40595 } } peers { permanent_uuid: "a46882de0a954f6b93c5449d6e09f340" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 39863 } } }
I20260808 04:03:59.364893 7728 tablet_replica.cc:442] T 323f5cf487fd417c940f794fed443f92 P 5fcb22667c684fa495415f7f391266c1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 04:03:59.366706 7759 tablet_replica.cc:439] T 323f5cf487fd417c940f794fed443f92 P 61a56ad7bdac46b199107e364bd82d4e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "61a56ad7bdac46b199107e364bd82d4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fcb22667c684fa495415f7f391266c1" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42131 } } peers { permanent_uuid: "61a56ad7bdac46b199107e364bd82d4e" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 40595 } } peers { permanent_uuid: "a46882de0a954f6b93c5449d6e09f340" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 39863 } } }
I20260808 04:03:59.366997 7759 tablet_replica.cc:442] T 323f5cf487fd417c940f794fed443f92 P 61a56ad7bdac46b199107e364bd82d4e: This TxnStatusTablet replica's current role is: LEADER
I20260808 04:03:59.369529 7728 tablet_replica.cc:439] T 323f5cf487fd417c940f794fed443f92 P 5fcb22667c684fa495415f7f391266c1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "61a56ad7bdac46b199107e364bd82d4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fcb22667c684fa495415f7f391266c1" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42131 } } peers { permanent_uuid: "61a56ad7bdac46b199107e364bd82d4e" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 40595 } } peers { permanent_uuid: "a46882de0a954f6b93c5449d6e09f340" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 39863 } } }
I20260808 04:03:59.369819 7728 tablet_replica.cc:442] T 323f5cf487fd417c940f794fed443f92 P 5fcb22667c684fa495415f7f391266c1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 04:03:59.382524 7789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:59.376602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.2:40595 (local address 127.5.83.1:52475)
0808 04:03:59.377186 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:59.377204 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 04:03:59.377320 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:03:59.377664 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:03:59.377667 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:03:59.377676 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 04:03:59.377886 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:59.377894 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:59.379306 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:59.379311 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:59.380472 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:59.380486 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:59.381441 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:59.381446 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:59.381557 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:59.381607 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 04:03:59.382003 (+ 396us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 04:03:59.382012 (+ 9us) client_negotiation.cc:769] Sending connection context
0808 04:03:59.382078 (+ 66us) client_negotiation.cc:240] Negotiation successful
0808 04:03:59.382179 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":295,"threads_started":1}
I20260808 04:03:59.382558 7768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:59.376711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.1:52475 (local address 127.5.83.2:40595)
0808 04:03:59.376887 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:59.376894 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:59.377284 (+ 390us) server_negotiation.cc:407] Connection header received
0808 04:03:59.377458 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:59.377464 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:59.377558 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:59.377645 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:03:59.378004 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:59.379162 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:59.380607 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:59.381259 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:59.381403 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:59.381763 (+ 360us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:03:59.381906 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:03:59.381992 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:59.382117 (+ 125us) server_negotiation.cc:299] Negotiation successful
0808 04:03:59.382200 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 04:03:59.392387 7592 raft_consensus.cc:1261] T 6553e5d9fcd041f480e89f8ef47d3ed0 P a46882de0a954f6b93c5449d6e09f340 [term 1 FOLLOWER]: Refusing update from remote peer 5fcb22667c684fa495415f7f391266c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:03:59.392756 7400 raft_consensus.cc:1261] T 6553e5d9fcd041f480e89f8ef47d3ed0 P 61a56ad7bdac46b199107e364bd82d4e [term 1 FOLLOWER]: Refusing update from remote peer 5fcb22667c684fa495415f7f391266c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:03:59.392993 7715 consensus_queue.cc:1035] T 6553e5d9fcd041f480e89f8ef47d3ed0 P 5fcb22667c684fa495415f7f391266c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a46882de0a954f6b93c5449d6e09f340" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:03:59.393404 7715 consensus_queue.cc:1035] T 6553e5d9fcd041f480e89f8ef47d3ed0 P 5fcb22667c684fa495415f7f391266c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61a56ad7bdac46b199107e364bd82d4e" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 40595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:03:59.399873 7774 mvcc.cc:205] Tried to move back new op lower bound from 7316118894146301952 to 7316118892737011712. Current Snapshot: MvccSnapshot[applied={T|T < 7316118894146301952}]
I20260808 04:03:59.455797 7794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:59.427151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:42131 (local address 127.5.83.2:54889)
0808 04:03:59.442687 (+ 15536us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:59.442705 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 04:03:59.442944 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:03:59.448523 (+ 5579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:03:59.448529 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:03:59.448544 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 04:03:59.448956 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:59.448969 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:59.451140 (+ 2171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:59.451146 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:59.453088 (+ 1942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:59.453104 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:59.454201 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:59.454207 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:59.454392 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:59.454466 (+ 74us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 04:03:59.454958 (+ 492us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 04:03:59.454970 (+ 12us) client_negotiation.cc:769] Sending connection context
0808 04:03:59.455085 (+ 115us) client_negotiation.cc:240] Negotiation successful
0808 04:03:59.455230 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15336,"thread_start_us":15175,"threads_started":1}
I20260808 04:03:59.456641 7793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:59.427150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.2:54889 (local address 127.5.83.1:42131)
0808 04:03:59.448201 (+ 21051us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:59.448208 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:59.448226 (+ 18us) server_negotiation.cc:407] Connection header received
0808 04:03:59.448293 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:59.448298 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:59.448374 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:59.449332 (+ 958us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:03:59.449586 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:59.450987 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:59.453242 (+ 2255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:59.454029 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:59.454155 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:59.454701 (+ 546us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:03:59.454846 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:03:59.454946 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:59.455109 (+ 163us) server_negotiation.cc:299] Negotiation successful
0808 04:03:59.455187 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20924,"thread_start_us":20794,"threads_started":1}
I20260808 04:03:59.491971 6986 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44018:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:03:59.492231 6986 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44018:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:03:59.495102 6986 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0f18f42ae17b4626a802d0e17926c51a: Sending DeleteTablet for 3 replicas of tablet 6553e5d9fcd041f480e89f8ef47d3ed0
I20260808 04:03:59.495790 7179 tablet_service.cc:1514] Processing DeleteTablet for tablet 6553e5d9fcd041f480e89f8ef47d3ed0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:03:59 UTC) from {username='kudu', principal='kudu/127.5.83.62@KRBTEST.COM'} at 127.0.0.1:39184
I20260808 04:03:59.496019 7562 tablet_service.cc:1514] Processing DeleteTablet for tablet 6553e5d9fcd041f480e89f8ef47d3ed0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:03:59 UTC) from {username='kudu', principal='kudu/127.5.83.62@KRBTEST.COM'} at 127.0.0.1:56462
I20260808 04:03:59.496629 7367 tablet_service.cc:1514] Processing DeleteTablet for tablet 6553e5d9fcd041f480e89f8ef47d3ed0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:03:59 UTC) from {username='kudu', principal='kudu/127.5.83.62@KRBTEST.COM'} at 127.0.0.1:56342
I20260808 04:03:59.505353 7799 tablet_replica.cc:330] T 6553e5d9fcd041f480e89f8ef47d3ed0 P a46882de0a954f6b93c5449d6e09f340: stopping tablet replica
I20260808 04:03:59.506078 7799 raft_consensus.cc:2229] T 6553e5d9fcd041f480e89f8ef47d3ed0 P a46882de0a954f6b93c5449d6e09f340 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:03:59.507113 7799 raft_consensus.cc:2258] T 6553e5d9fcd041f480e89f8ef47d3ed0 P a46882de0a954f6b93c5449d6e09f340 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:03:59.508387 7799 ts_tablet_manager.cc:1905] T 6553e5d9fcd041f480e89f8ef47d3ed0 P a46882de0a954f6b93c5449d6e09f340: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:03:59.512928 7800 tablet_replica.cc:330] T 6553e5d9fcd041f480e89f8ef47d3ed0 P 61a56ad7bdac46b199107e364bd82d4e: stopping tablet replica
I20260808 04:03:59.513145 7800 raft_consensus.cc:2229] T 6553e5d9fcd041f480e89f8ef47d3ed0 P 61a56ad7bdac46b199107e364bd82d4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:03:59.513434 7800 raft_consensus.cc:2258] T 6553e5d9fcd041f480e89f8ef47d3ed0 P 61a56ad7bdac46b199107e364bd82d4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:03:59.514627 7800 ts_tablet_manager.cc:1905] T 6553e5d9fcd041f480e89f8ef47d3ed0 P 61a56ad7bdac46b199107e364bd82d4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:03:59.514887 7798 tablet_replica.cc:330] T 6553e5d9fcd041f480e89f8ef47d3ed0 P 5fcb22667c684fa495415f7f391266c1: stopping tablet replica
I20260808 04:03:59.515082 7798 raft_consensus.cc:2229] T 6553e5d9fcd041f480e89f8ef47d3ed0 P 5fcb22667c684fa495415f7f391266c1 [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:03:59.515520 7798 raft_consensus.cc:2258] T 6553e5d9fcd041f480e89f8ef47d3ed0 P 5fcb22667c684fa495415f7f391266c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:03:59.516605 7798 ts_tablet_manager.cc:1905] T 6553e5d9fcd041f480e89f8ef47d3ed0 P 5fcb22667c684fa495415f7f391266c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:03:59.519474 7799 ts_tablet_manager.cc:1918] T 6553e5d9fcd041f480e89f8ef47d3ed0 P a46882de0a954f6b93c5449d6e09f340: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 04:03:59.522346 7800 ts_tablet_manager.cc:1918] T 6553e5d9fcd041f480e89f8ef47d3ed0 P 61a56ad7bdac46b199107e364bd82d4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 04:03:59.522481 7800 log.cc:1196] T 6553e5d9fcd041f480e89f8ef47d3ed0 P 61a56ad7bdac46b199107e364bd82d4e: Deleting WAL directory at /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/minicluster-data/ts-1/wal/wals/6553e5d9fcd041f480e89f8ef47d3ed0
I20260808 04:03:59.522607 7799 log.cc:1196] T 6553e5d9fcd041f480e89f8ef47d3ed0 P a46882de0a954f6b93c5449d6e09f340: Deleting WAL directory at /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/minicluster-data/ts-2/wal/wals/6553e5d9fcd041f480e89f8ef47d3ed0
I20260808 04:03:59.523057 7800 ts_tablet_manager.cc:1939] T 6553e5d9fcd041f480e89f8ef47d3ed0 P 61a56ad7bdac46b199107e364bd82d4e: Deleting consensus metadata
I20260808 04:03:59.523259 7799 ts_tablet_manager.cc:1939] T 6553e5d9fcd041f480e89f8ef47d3ed0 P a46882de0a954f6b93c5449d6e09f340: Deleting consensus metadata
I20260808 04:03:59.523533 7798 ts_tablet_manager.cc:1918] T 6553e5d9fcd041f480e89f8ef47d3ed0 P 5fcb22667c684fa495415f7f391266c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 04:03:59.523664 7798 log.cc:1196] T 6553e5d9fcd041f480e89f8ef47d3ed0 P 5fcb22667c684fa495415f7f391266c1: Deleting WAL directory at /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161826605655-5452-0/minicluster-data/ts-0/wal/wals/6553e5d9fcd041f480e89f8ef47d3ed0
I20260808 04:03:59.524298 7798 ts_tablet_manager.cc:1939] T 6553e5d9fcd041f480e89f8ef47d3ed0 P 5fcb22667c684fa495415f7f391266c1: Deleting consensus metadata
I20260808 04:03:59.524240 6967 catalog_manager.cc:4841] TS 61a56ad7bdac46b199107e364bd82d4e (127.5.83.2:40595): tablet 6553e5d9fcd041f480e89f8ef47d3ed0 (table test-table [id=cd5d25cb18dd40e0a6a0dbdde957be3e]) successfully deleted
I20260808 04:03:59.525364 6964 catalog_manager.cc:4841] TS 5fcb22667c684fa495415f7f391266c1 (127.5.83.1:42131): tablet 6553e5d9fcd041f480e89f8ef47d3ed0 (table test-table [id=cd5d25cb18dd40e0a6a0dbdde957be3e]) successfully deleted
I20260808 04:03:59.525686 6967 catalog_manager.cc:4841] TS a46882de0a954f6b93c5449d6e09f340 (127.5.83.3:39863): tablet 6553e5d9fcd041f480e89f8ef47d3ed0 (table test-table [id=cd5d25cb18dd40e0a6a0dbdde957be3e]) successfully deleted
Aug 08 04:03:59 dist-test-slave-1l3l krb5kdc[6923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161838, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.83.1@KRBTEST.COM
I20260808 04:03:59.536291 7793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:59.507229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39208 (local address 127.5.83.1:42131)
0808 04:03:59.507389 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:59.507396 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:59.513808 (+ 6412us) server_negotiation.cc:407] Connection header received
0808 04:03:59.514697 (+ 889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:59.514703 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:59.514792 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:59.514868 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:59.519668 (+ 4800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:59.520756 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:59.523937 (+ 3181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:59.524669 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:59.524766 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:59.529898 (+ 5132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:59.529913 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:59.529930 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:03:59.529974 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:59.533620 (+ 3646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:59.534272 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:59.534283 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:59.534286 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:59.534470 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:59.534793 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:59.534798 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:59.534801 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:59.535202 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:59.535531 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:59.535686 (+ 155us) server_negotiation.cc:299] Negotiation successful
0808 04:03:59.535938 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
W20260808 04:03:59.537065 7268 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39208
I20260808 04:03:59.557783 7751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:59.544818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44040 (local address 127.5.83.62:45699)
0808 04:03:59.545019 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:59.545025 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:59.545186 (+ 161us) server_negotiation.cc:407] Connection header received
0808 04:03:59.545303 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:59.545308 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:59.545459 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:59.545567 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:59.546760 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:59.547916 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:59.549049 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:59.549696 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:59.549819 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:59.551943 (+ 2124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:59.551949 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:59.551955 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:03:59.551994 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:59.555266 (+ 3272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:59.555979 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:59.555985 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:59.555989 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:59.556133 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:59.556412 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:59.556416 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:59.556420 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:59.556768 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:59.557000 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:59.557248 (+ 248us) server_negotiation.cc:299] Negotiation successful
0808 04:03:59.557469 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
W20260808 04:03:59.558373 6986 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44040
I20260808 04:03:59.560797 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 7084
I20260808 04:03:59.582911 7793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:59.561490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39224 (local address 127.5.83.1:42131)
0808 04:03:59.561700 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:59.561707 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:59.570269 (+ 8562us) server_negotiation.cc:407] Connection header received
0808 04:03:59.570406 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:59.570412 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:59.570495 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:59.570604 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:59.571739 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:59.572916 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:59.574088 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:59.574725 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:59.574858 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:59.576945 (+ 2087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:59.576952 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:59.576956 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:03:59.577000 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:59.580294 (+ 3294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:59.581038 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:59.581066 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:59.581069 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:59.581224 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:59.581564 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:59.581569 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:59.581572 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:59.581909 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:59.582163 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:59.582322 (+ 159us) server_negotiation.cc:299] Negotiation successful
0808 04:03:59.582551 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260808 04:03:59.583536 7268 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39224
W20260808 04:03:59.584080 5452 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 7084: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 04:03:59.584249 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 7084
I20260808 04:03:59.623246 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 7279
Aug 08 04:03:59 dist-test-slave-1l3l krb5kdc[6923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161838, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.83.2@KRBTEST.COM
I20260808 04:03:59.639617 7768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:59.624240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56362 (local address 127.5.83.2:40595)
0808 04:03:59.624465 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:59.624471 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:59.624502 (+ 31us) server_negotiation.cc:407] Connection header received
0808 04:03:59.624570 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:59.624575 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:59.624656 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:59.624790 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:59.626017 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:59.627155 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:59.628619 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:59.629280 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:59.629408 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:59.633664 (+ 4256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:59.633675 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:59.633691 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:03:59.633733 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:59.636856 (+ 3123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:59.637655 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:59.637665 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:59.637668 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:59.637806 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:59.638205 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:59.638208 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:59.638210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:59.638546 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:59.638779 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:59.639124 (+ 345us) server_negotiation.cc:299] Negotiation successful
0808 04:03:59.639294 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260808 04:03:59.640363 7459 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56362
W20260808 04:03:59.641026 5452 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 7279: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 04:03:59.641211 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 7279
I20260808 04:03:59.664887 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 7470
Aug 08 04:03:59 dist-test-slave-1l3l krb5kdc[6923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161838, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.83.3@KRBTEST.COM
I20260808 04:03:59.679050 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:59.665616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56474 (local address 127.5.83.3:39863)
0808 04:03:59.666021 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:59.666025 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:59.666042 (+ 17us) server_negotiation.cc:407] Connection header received
0808 04:03:59.666094 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:59.666098 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:59.666163 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:59.666252 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:59.667442 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:59.668169 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:59.669313 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:59.669725 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:59.669815 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:59.673804 (+ 3989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:59.673819 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:59.673836 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:03:59.673869 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:59.676539 (+ 2670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:59.677135 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:59.677143 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:59.677146 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:59.677357 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:59.677749 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:59.677752 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:59.677754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:59.678090 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:59.678320 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:59.678550 (+ 230us) server_negotiation.cc:299] Negotiation successful
0808 04:03:59.678717 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":203,"threads_started":1}
W20260808 04:03:59.679662 7654 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56474
W20260808 04:03:59.680337 5452 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 7470: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 04:03:59.680495 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 7470
I20260808 04:03:59.702131 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 6942
I20260808 04:03:59.703217 7041 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:03:59.815847 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 6942
2026-08-08T04:03:59Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4969 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:03:59 dist-test-slave-1l3l krb5kdc[7810](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:03:59 dist-test-slave-1l3l krb5kdc[7810](info): set up 2 sockets
Aug 08 04:03:59 dist-test-slave-1l3l krb5kdc[7810](info): commencing operation
krb5kdc: starting...
W20260808 04:04:00.986539 5452 mini_kdc.cc:121] Time spent starting KDC: real 1.126s user 0.001s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/krb5kdc/test-user.keytab.
Aug 08 04:04:01 dist-test-slave-1l3l krb5kdc[7810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:04:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:04:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.83.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab.
Entry for principal kudu/127.5.83.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab.
I20260808 04:04:01.173666 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:45715
--webserver_interface=127.5.83.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab
--principal=kudu/127.5.83.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:33645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.62:45715
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:04:01.487610 7834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:04:01.487951 7834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:04:01.488092 7834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:04:01.495085 7834 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:04:01.495185 7834 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260808 04:04:01.495252 7834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:04:01.495350 7834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:04:01.495422 7834 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:04:01.495505 7834 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:04:01.504889 7834 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:33645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.62:45715
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab
--principal=kudu/127.5.83.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:45715
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:04:01.506021 7834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:04:01.507222 7834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:04:01.518065 7847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:01.518146 7843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:01.518031 7844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:04:01.518774 7834 server_base.cc:1034] running on GCE node
I20260808 04:04:01.519439 7834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:04:01.520681 7834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:04:01.521880 7834 hybrid_clock.cc:648] HybridClock initialized: now 1786161841521822 us; error 33 us; skew 500 ppm
I20260808 04:04:01.522166 7834 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:04:01 dist-test-slave-1l3l krb5kdc[7810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161841, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:04:01.525118 7834 init.cc:377] Logged in from keytab as kudu/127.5.83.62@KRBTEST.COM (short username kudu)
I20260808 04:04:01.526413 7834 webserver.cc:460] Webserver started at http://127.5.83.62:42603/ using document root <none> and password file <none>
I20260808 04:04:01.527021 7834 fs_manager.cc:359] Metadata directory not provided
I20260808 04:04:01.527109 7834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:04:01.527393 7834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:04:01.530292 7834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/minicluster-data/master-0/data/instance:
uuid: "4f45675d9036448fbf2bbd0b2d808bc7"
format_stamp: "Formatted at 2026-08-08 04:04:01 on dist-test-slave-1l3l"
server_key: "7b750eb44e113d2685977e22691f6862"
server_key_iv: "7706f11ee416a8e84c35801e129847ab"
server_key_version: "encryptionkey@0"
I20260808 04:04:01.531023 7834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/minicluster-data/master-0/wal/instance:
uuid: "4f45675d9036448fbf2bbd0b2d808bc7"
format_stamp: "Formatted at 2026-08-08 04:04:01 on dist-test-slave-1l3l"
server_key: "7b750eb44e113d2685977e22691f6862"
server_key_iv: "7706f11ee416a8e84c35801e129847ab"
server_key_version: "encryptionkey@0"
I20260808 04:04:01.535789 7834 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260808 04:04:01.538838 7856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:01.540084 7834 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:04:01.540235 7834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/minicluster-data/master-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/minicluster-data/master-0/wal
uuid: "4f45675d9036448fbf2bbd0b2d808bc7"
format_stamp: "Formatted at 2026-08-08 04:04:01 on dist-test-slave-1l3l"
server_key: "7b750eb44e113d2685977e22691f6862"
server_key_iv: "7706f11ee416a8e84c35801e129847ab"
server_key_version: "encryptionkey@0"
I20260808 04:04:01.540429 7834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:04:01.566510 7834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:04:01.570508 7834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:04:01.570750 7834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:04:01.590256 7834 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.62:45715
I20260808 04:04:01.590272 7931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.62:45715 every 8 connection(s)
I20260808 04:04:01.591532 7834 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
I20260808 04:04:01.594980 7932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:04:01.600832 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 7834
I20260808 04:04:01.601030 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/minicluster-data/master-0/wal/instance
I20260808 04:04:01.601639 5452 external_mini_cluster.cc:1428] Setting key 515f249e643b170cafbd540843354248
I20260808 04:04:01.605255 7932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f45675d9036448fbf2bbd0b2d808bc7: Bootstrap starting.
I20260808 04:04:01.608783 7932 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f45675d9036448fbf2bbd0b2d808bc7: Neither blocks nor log segments found. Creating new log.
I20260808 04:04:01.610105 7932 log.cc:824] T 00000000000000000000000000000000 P 4f45675d9036448fbf2bbd0b2d808bc7: Log is configured to *not* fsync() on all Append() calls
I20260808 04:04:01.612977 7932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f45675d9036448fbf2bbd0b2d808bc7: No bootstrap required, opened a new log
Aug 08 04:04:01 dist-test-slave-1l3l krb5kdc[7810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:04:01.620740 7932 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f45675d9036448fbf2bbd0b2d808bc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f45675d9036448fbf2bbd0b2d808bc7" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 45715 } }
I20260808 04:04:01.621162 7932 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4f45675d9036448fbf2bbd0b2d808bc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:04:01.621270 7932 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f45675d9036448fbf2bbd0b2d808bc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f45675d9036448fbf2bbd0b2d808bc7, State: Initialized, Role: FOLLOWER
I20260808 04:04:01.621836 7932 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f45675d9036448fbf2bbd0b2d808bc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f45675d9036448fbf2bbd0b2d808bc7" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 45715 } }
I20260808 04:04:01.622054 7932 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4f45675d9036448fbf2bbd0b2d808bc7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:04:01.622191 7932 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4f45675d9036448fbf2bbd0b2d808bc7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:04:01.622349 7932 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f45675d9036448fbf2bbd0b2d808bc7 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:04:01.624727 7938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:01.603511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48892 (local address 127.5.83.62:45715)
0808 04:04:01.604348 (+ 837us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:01.604362 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:01.604410 (+ 48us) server_negotiation.cc:407] Connection header received
0808 04:04:01.606636 (+ 2226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:01.606662 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:01.607158 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:01.607562 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:01.608927 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:01.610874 (+ 1947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:01.612219 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:01.613175 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:01.613354 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:01.617491 (+ 4137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:01.617498 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:01.617511 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:01.617544 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:01.620502 (+ 2958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:01.621342 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:01.621349 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:01.621357 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:01.621533 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:01.621889 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:01.621893 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:01.621895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:01.622487 (+ 592us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:01.622767 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:01.623245 (+ 478us) server_negotiation.cc:299] Negotiation successful
0808 04:04:01.623641 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":378,"threads_started":1}
I20260808 04:04:01.624694 7932 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4f45675d9036448fbf2bbd0b2d808bc7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f45675d9036448fbf2bbd0b2d808bc7" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 45715 } }
I20260808 04:04:01.625165 7932 leader_election.cc:302] T 00000000000000000000000000000000 P 4f45675d9036448fbf2bbd0b2d808bc7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4f45675d9036448fbf2bbd0b2d808bc7; no voters:
I20260808 04:04:01.625759 7932 leader_election.cc:288] T 00000000000000000000000000000000 P 4f45675d9036448fbf2bbd0b2d808bc7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:04:01.625870 7941 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4f45675d9036448fbf2bbd0b2d808bc7 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:04:01.626957 7941 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4f45675d9036448fbf2bbd0b2d808bc7 [term 1 LEADER]: Becoming Leader. State: Replica: 4f45675d9036448fbf2bbd0b2d808bc7, State: Running, Role: LEADER
I20260808 04:04:01.627519 7941 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f45675d9036448fbf2bbd0b2d808bc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f45675d9036448fbf2bbd0b2d808bc7" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 45715 } }
I20260808 04:04:01.628010 7932 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f45675d9036448fbf2bbd0b2d808bc7 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:04:01.630311 7942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f45675d9036448fbf2bbd0b2d808bc7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4f45675d9036448fbf2bbd0b2d808bc7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f45675d9036448fbf2bbd0b2d808bc7" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 45715 } } }
I20260808 04:04:01.630678 7942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f45675d9036448fbf2bbd0b2d808bc7 [sys.catalog]: This master's current role is: LEADER
I20260808 04:04:01.635135 7943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f45675d9036448fbf2bbd0b2d808bc7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f45675d9036448fbf2bbd0b2d808bc7. Latest consensus state: current_term: 1 leader_uuid: "4f45675d9036448fbf2bbd0b2d808bc7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f45675d9036448fbf2bbd0b2d808bc7" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 45715 } } }
I20260808 04:04:01.635514 7943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f45675d9036448fbf2bbd0b2d808bc7 [sys.catalog]: This master's current role is: LEADER
I20260808 04:04:01.635967 7946 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:04:01.640502 7946 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:04:01.648350 7946 catalog_manager.cc:1347] Generated new cluster ID: fa8511b30c9c4a4d8e57e3ef77b7062e
I20260808 04:04:01.648465 7946 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:04:01.671401 7946 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:04:01.673185 7946 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:04:01.684587 7946 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4f45675d9036448fbf2bbd0b2d808bc7: Generated new TSK 0
I20260808 04:04:01.685544 7946 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.83.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.1.keytab.
Entry for principal kudu/127.5.83.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.1.keytab.
I20260808 04:04:01.730260 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.1:0
--local_ip_for_outbound_sockets=127.5.83.1
--webserver_interface=127.5.83.1
--webserver_port=0
--tserver_master_addrs=127.5.83.62:45715
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.1.keytab
--principal=kudu/127.5.83.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:33645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:04:02.041558 7969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:04:02.041811 7969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:04:02.041909 7969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:04:02.048887 7969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:04:02.048981 7969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:04:02.049069 7969 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260808 04:04:02.049119 7969 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 04:04:02.049185 7969 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.1
I20260808 04:04:02.058416 7969 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:33645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.1.keytab
--principal=kudu/127.5.83.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.83.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.83.62:45715
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.1
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:04:02.059546 7969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:04:02.060856 7969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:04:02.072804 7976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:02.072867 7977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:02.072867 7980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:04:02.073856 7969 server_base.cc:1034] running on GCE node
I20260808 04:04:02.074437 7969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:04:02.075320 7969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:04:02.076555 7969 hybrid_clock.cc:648] HybridClock initialized: now 1786161842076504 us; error 40 us; skew 500 ppm
I20260808 04:04:02.076844 7969 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:04:02 dist-test-slave-1l3l krb5kdc[7810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161842, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:04:02.079795 7969 init.cc:377] Logged in from keytab as kudu/127.5.83.1@KRBTEST.COM (short username kudu)
I20260808 04:04:02.081063 7969 webserver.cc:460] Webserver started at http://127.5.83.1:32909/ using document root <none> and password file <none>
I20260808 04:04:02.081673 7969 fs_manager.cc:359] Metadata directory not provided
I20260808 04:04:02.081784 7969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:04:02.082070 7969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:04:02.084857 7969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/minicluster-data/ts-0/data/instance:
uuid: "ac14eb089ed14bdbbd682f20cb7e9ada"
format_stamp: "Formatted at 2026-08-08 04:04:02 on dist-test-slave-1l3l"
server_key: "62d9fc57d33e876ad99954c753332d4f"
server_key_iv: "62e28a426a121d08ff30dbe41a6b40a5"
server_key_version: "encryptionkey@0"
I20260808 04:04:02.085569 7969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/minicluster-data/ts-0/wal/instance:
uuid: "ac14eb089ed14bdbbd682f20cb7e9ada"
format_stamp: "Formatted at 2026-08-08 04:04:02 on dist-test-slave-1l3l"
server_key: "62d9fc57d33e876ad99954c753332d4f"
server_key_iv: "62e28a426a121d08ff30dbe41a6b40a5"
server_key_version: "encryptionkey@0"
I20260808 04:04:02.090359 7969 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260808 04:04:02.093812 7989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:02.095067 7969 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 04:04:02.095203 7969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/minicluster-data/ts-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/minicluster-data/ts-0/wal
uuid: "ac14eb089ed14bdbbd682f20cb7e9ada"
format_stamp: "Formatted at 2026-08-08 04:04:02 on dist-test-slave-1l3l"
server_key: "62d9fc57d33e876ad99954c753332d4f"
server_key_iv: "62e28a426a121d08ff30dbe41a6b40a5"
server_key_version: "encryptionkey@0"
I20260808 04:04:02.095438 7969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:04:02.109300 7969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:04:02.112895 7969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:04:02.113142 7969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:04:02.115631 7969 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:04:02.115790 7969 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260808 04:04:02.115909 7969 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:04:02.116006 7969 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 04:04:02 dist-test-slave-1l3l krb5kdc[7810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161842, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.1@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:04:02.137691 7938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:02.116202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.1:33541 (local address 127.5.83.62:45715)
0808 04:04:02.116385 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:02.116392 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:02.117703 (+ 1311us) server_negotiation.cc:407] Connection header received
0808 04:04:02.119749 (+ 2046us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:02.119755 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:02.119856 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:02.119969 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:02.122061 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:02.123038 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:02.125836 (+ 2798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:02.126348 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:02.126470 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:02.131141 (+ 4671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:02.131146 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:02.131149 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:02.131183 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:02.134371 (+ 3188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:02.135244 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:02.135251 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:02.135255 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:02.135431 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:02.135974 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:02.135979 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:02.135982 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:02.136279 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:02.136394 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:02.137241 (+ 847us) server_negotiation.cc:299] Negotiation successful
0808 04:04:02.137425 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 04:04:02.139148 7998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:02.116750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:45715 (local address 127.5.83.1:33541)
0808 04:04:02.117552 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:02.117587 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:02.119388 (+ 1801us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:02.120238 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:02.120253 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:02.121087 (+ 834us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:04:02.121889 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:02.121910 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:02.123227 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:02.123233 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:02.125668 (+ 2435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:02.125685 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:02.126602 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:02.126614 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:02.126816 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:02.127657 (+ 841us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:04:02.127731 (+ 74us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:04:02.130926 (+ 3195us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:04:02.134517 (+ 3591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:02.134526 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:02.134540 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:02.135112 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:02.135544 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:02.135548 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:02.135550 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:02.135781 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:02.136409 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:04:02.136417 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:04:02.136852 (+ 435us) client_negotiation.cc:769] Sending connection context
0808 04:04:02.137160 (+ 308us) client_negotiation.cc:240] Negotiation successful
0808 04:04:02.137695 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":379,"threads_started":1}
I20260808 04:04:02.158996 7938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:02.145177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.1:53299 (local address 127.5.83.62:45715)
0808 04:04:02.145349 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:02.145355 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:02.145804 (+ 449us) server_negotiation.cc:407] Connection header received
0808 04:04:02.145962 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:02.145965 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:02.146025 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:02.146103 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:02.147227 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:02.148038 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:02.149924 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:02.150468 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:02.150576 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:02.152942 (+ 2366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:02.152949 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:02.152953 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:02.152989 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:02.156346 (+ 3357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:02.157174 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:02.157178 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:02.157180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:02.157298 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:02.157639 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:02.157644 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:02.157646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:02.157901 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:02.158032 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:02.158443 (+ 411us) server_negotiation.cc:299] Negotiation successful
0808 04:04:02.158648 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 04:04:02.159088 8083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:02.144862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:45715 (local address 127.5.83.1:53299)
0808 04:04:02.145698 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:02.145718 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:02.145867 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:02.146141 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:02.146146 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:02.146731 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:04:02.147108 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:02.147120 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:02.148184 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:02.148190 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:02.149787 (+ 1597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:02.149800 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:02.150651 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:02.150657 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:02.150818 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:02.151443 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:04:02.151466 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:04:02.152708 (+ 1242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:04:02.156475 (+ 3767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:02.156479 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:02.156482 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:02.157011 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:02.157404 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:02.157410 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:02.157414 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:02.157533 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:02.158014 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:04:02.158019 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:04:02.158190 (+ 171us) client_negotiation.cc:769] Sending connection context
0808 04:04:02.158322 (+ 132us) client_negotiation.cc:240] Negotiation successful
0808 04:04:02.158639 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":680,"thread_start_us":542,"threads_started":1}
I20260808 04:04:02.169790 7969 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.1:42739
I20260808 04:04:02.169790 8158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.1:42739 every 8 connection(s)
I20260808 04:04:02.170974 7969 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
I20260808 04:04:02.172798 8159 heartbeater.cc:347] Connected to a master server at 127.5.83.62:45715
I20260808 04:04:02.172988 8159 heartbeater.cc:464] Registering TS with master...
I20260808 04:04:02.173547 8159 heartbeater.cc:511] Master 127.5.83.62:45715 requested a full tablet report, sending...
I20260808 04:04:02.175284 7875 ts_manager.cc:194] Registered new tserver with Master: ac14eb089ed14bdbbd682f20cb7e9ada (127.5.83.1:42739)
I20260808 04:04:02.176247 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 7969
I20260808 04:04:02.176429 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/minicluster-data/ts-0/wal/instance
I20260808 04:04:02.176792 7875 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.1@KRBTEST.COM'} at 127.5.83.1:33541
I20260808 04:04:02.177060 5452 external_mini_cluster.cc:1428] Setting key 48f3d67df914ad40f3b37eed79190765
WARNING: no policy specified for kudu/127.5.83.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.2.keytab.
Entry for principal kudu/127.5.83.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.2.keytab.
I20260808 04:04:02.215992 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.2:0
--local_ip_for_outbound_sockets=127.5.83.2
--webserver_interface=127.5.83.2
--webserver_port=0
--tserver_master_addrs=127.5.83.62:45715
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.2.keytab
--principal=kudu/127.5.83.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:33645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:04:02.531822 8164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:04:02.532068 8164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:04:02.532203 8164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:04:02.539144 8164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:04:02.539244 8164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:04:02.539372 8164 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260808 04:04:02.539436 8164 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 04:04:02.539532 8164 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.2
I20260808 04:04:02.548971 8164 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:33645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.2.keytab
--principal=kudu/127.5.83.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.83.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.83.62:45715
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.2
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:04:02.550103 8164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:04:02.551321 8164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:04:02.563158 8170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:02.563233 8173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:02.563227 8171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:04:02.564368 8164 server_base.cc:1034] running on GCE node
I20260808 04:04:02.564911 8164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:04:02.565721 8164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:04:02.566927 8164 hybrid_clock.cc:648] HybridClock initialized: now 1786161842566872 us; error 38 us; skew 500 ppm
I20260808 04:04:02.567214 8164 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:04:02 dist-test-slave-1l3l krb5kdc[7810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161842, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:04:02.570188 8164 init.cc:377] Logged in from keytab as kudu/127.5.83.2@KRBTEST.COM (short username kudu)
I20260808 04:04:02.571585 8164 webserver.cc:460] Webserver started at http://127.5.83.2:45677/ using document root <none> and password file <none>
I20260808 04:04:02.572243 8164 fs_manager.cc:359] Metadata directory not provided
I20260808 04:04:02.572364 8164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:04:02.572654 8164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:04:02.575546 8164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/minicluster-data/ts-1/data/instance:
uuid: "1f623bcbb73947e8ba2becbff3aa228f"
format_stamp: "Formatted at 2026-08-08 04:04:02 on dist-test-slave-1l3l"
server_key: "e4b9e5a319d59f96dd5f180520bde0ba"
server_key_iv: "45551bea426e8c21d86e5613b8c38e9f"
server_key_version: "encryptionkey@0"
I20260808 04:04:02.576315 8164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/minicluster-data/ts-1/wal/instance:
uuid: "1f623bcbb73947e8ba2becbff3aa228f"
format_stamp: "Formatted at 2026-08-08 04:04:02 on dist-test-slave-1l3l"
server_key: "e4b9e5a319d59f96dd5f180520bde0ba"
server_key_iv: "45551bea426e8c21d86e5613b8c38e9f"
server_key_version: "encryptionkey@0"
I20260808 04:04:02.580996 8164 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260808 04:04:02.584262 8182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:02.585505 8164 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:04:02.585649 8164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/minicluster-data/ts-1/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/minicluster-data/ts-1/wal
uuid: "1f623bcbb73947e8ba2becbff3aa228f"
format_stamp: "Formatted at 2026-08-08 04:04:02 on dist-test-slave-1l3l"
server_key: "e4b9e5a319d59f96dd5f180520bde0ba"
server_key_iv: "45551bea426e8c21d86e5613b8c38e9f"
server_key_version: "encryptionkey@0"
I20260808 04:04:02.585815 8164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:04:02.601603 8164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:04:02.605597 8164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:04:02.605880 8164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:04:02.608515 8164 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:04:02.608631 8164 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:02.608742 8164 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:04:02.608839 8164 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 04:04:02 dist-test-slave-1l3l krb5kdc[7810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161842, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.2@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:04:02.629827 7938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:02.609035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.2:46291 (local address 127.5.83.62:45715)
0808 04:04:02.609300 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:02.609307 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:02.610763 (+ 1456us) server_negotiation.cc:407] Connection header received
0808 04:04:02.612786 (+ 2023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:02.612792 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:02.612880 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:02.612969 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:02.614938 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:02.615964 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:02.618337 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:02.618968 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:02.619108 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:02.623079 (+ 3971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:02.623087 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:02.623091 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:02.623140 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:02.626417 (+ 3277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:02.627091 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:02.627097 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:02.627101 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:02.627223 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:02.627631 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:02.627636 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:02.627639 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:02.627993 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:02.628166 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:02.629222 (+ 1056us) server_negotiation.cc:299] Negotiation successful
0808 04:04:02.629476 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20260808 04:04:02.631217 8196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:02.609663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:45715 (local address 127.5.83.2:46291)
0808 04:04:02.610610 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:02.610646 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:02.612525 (+ 1879us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:02.613214 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:02.613222 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:02.613930 (+ 708us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:04:02.614765 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:02.614790 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:02.616117 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:02.616122 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:02.618164 (+ 2042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:02.618184 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:02.619181 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:02.619186 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:02.619389 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:02.620070 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:04:02.620101 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:04:02.622847 (+ 2746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:04:02.626550 (+ 3703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:02.626558 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:02.626569 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:02.626977 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:02.627326 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:02.627331 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:02.627335 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:02.627520 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:02.628251 (+ 731us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:04:02.628259 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:04:02.628771 (+ 512us) client_negotiation.cc:769] Sending connection context
0808 04:04:02.629126 (+ 355us) client_negotiation.cc:240] Negotiation successful
0808 04:04:02.629704 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":731,"thread_start_us":471,"threads_started":1}
I20260808 04:04:02.649514 7938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:02.636458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.2:38693 (local address 127.5.83.62:45715)
0808 04:04:02.636634 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:02.636640 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:02.637488 (+ 848us) server_negotiation.cc:407] Connection header received
0808 04:04:02.637675 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:02.637680 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:02.637745 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:02.637820 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:02.638722 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:02.639453 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:02.641461 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:02.641981 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:02.642053 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:02.643945 (+ 1892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:02.643951 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:02.643956 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:02.643986 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:02.646787 (+ 2801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:02.647569 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:02.647574 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:02.647577 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:02.647753 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:02.648229 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:02.648234 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:02.648237 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:02.648578 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:02.648690 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:02.649098 (+ 408us) server_negotiation.cc:299] Negotiation successful
0808 04:04:02.649267 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260808 04:04:02.649850 8282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:02.636540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:45715 (local address 127.5.83.2:38693)
0808 04:04:02.637395 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:02.637415 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:02.637558 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:02.637870 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:02.637875 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:02.638299 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:04:02.638588 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:02.638597 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:02.639584 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:02.639590 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:02.641284 (+ 1694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:02.641300 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:02.642266 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:02.642270 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:02.642394 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:02.642956 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:04:02.642980 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:04:02.643770 (+ 790us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:04:02.646908 (+ 3138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:02.646914 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:02.646917 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:02.647437 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:02.647912 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:02.647916 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:02.647920 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:02.648099 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:02.648675 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:04:02.648680 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:04:02.648876 (+ 196us) client_negotiation.cc:769] Sending connection context
0808 04:04:02.649043 (+ 167us) client_negotiation.cc:240] Negotiation successful
0808 04:04:02.649371 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":699,"thread_start_us":464,"threads_started":1}
I20260808 04:04:02.661211 8164 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.2:41193
I20260808 04:04:02.661248 8343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.2:41193 every 8 connection(s)
I20260808 04:04:02.662369 8164 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
I20260808 04:04:02.664382 8345 heartbeater.cc:347] Connected to a master server at 127.5.83.62:45715
I20260808 04:04:02.664593 8345 heartbeater.cc:464] Registering TS with master...
I20260808 04:04:02.665192 8345 heartbeater.cc:511] Master 127.5.83.62:45715 requested a full tablet report, sending...
I20260808 04:04:02.666460 7875 ts_manager.cc:194] Registered new tserver with Master: 1f623bcbb73947e8ba2becbff3aa228f (127.5.83.2:41193)
I20260808 04:04:02.667248 7875 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.2@KRBTEST.COM'} at 127.5.83.2:46291
I20260808 04:04:02.672286 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 8164
I20260808 04:04:02.672497 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/minicluster-data/ts-1/wal/instance
I20260808 04:04:02.673015 5452 external_mini_cluster.cc:1428] Setting key ce93cf8933ffb5bcf775322f0a97ca90
WARNING: no policy specified for kudu/127.5.83.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.3.keytab.
Entry for principal kudu/127.5.83.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.3.keytab.
I20260808 04:04:02.712989 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.3:0
--local_ip_for_outbound_sockets=127.5.83.3
--webserver_interface=127.5.83.3
--webserver_port=0
--tserver_master_addrs=127.5.83.62:45715
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.3.keytab
--principal=kudu/127.5.83.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:33645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:04:03.026366 8349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:04:03.026633 8349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:04:03.026791 8349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:04:03.036883 8349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:04:03.037000 8349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:04:03.037122 8349 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260808 04:04:03.037178 8349 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 04:04:03.037245 8349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.3
I20260808 04:04:03.046710 8349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:33645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.3.keytab
--principal=kudu/127.5.83.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.83.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.83.62:45715
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.3
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:04:03.047883 8349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:04:03.049167 8349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:04:03.061510 8357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:03.061542 8355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:03.061563 8359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:04:03.062213 8349 server_base.cc:1034] running on GCE node
I20260808 04:04:03.062750 8349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:04:03.063586 8349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:04:03.064774 8349 hybrid_clock.cc:648] HybridClock initialized: now 1786161843064736 us; error 50 us; skew 500 ppm
I20260808 04:04:03.065037 8349 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:04:03 dist-test-slave-1l3l krb5kdc[7810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161843, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:04:03.068059 8349 init.cc:377] Logged in from keytab as kudu/127.5.83.3@KRBTEST.COM (short username kudu)
I20260808 04:04:03.069708 8349 webserver.cc:460] Webserver started at http://127.5.83.3:33341/ using document root <none> and password file <none>
I20260808 04:04:03.070326 8349 fs_manager.cc:359] Metadata directory not provided
I20260808 04:04:03.070439 8349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:04:03.070737 8349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:04:03.073487 8349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/minicluster-data/ts-2/data/instance:
uuid: "09ea3124898843e0a7a68d99d83c9f83"
format_stamp: "Formatted at 2026-08-08 04:04:03 on dist-test-slave-1l3l"
server_key: "7c278e92cc735a9d3f3be1ce95d83c3c"
server_key_iv: "27a5bbac9f9064a6af3efb1a971da400"
server_key_version: "encryptionkey@0"
I20260808 04:04:03.074199 8349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/minicluster-data/ts-2/wal/instance:
uuid: "09ea3124898843e0a7a68d99d83c9f83"
format_stamp: "Formatted at 2026-08-08 04:04:03 on dist-test-slave-1l3l"
server_key: "7c278e92cc735a9d3f3be1ce95d83c3c"
server_key_iv: "27a5bbac9f9064a6af3efb1a971da400"
server_key_version: "encryptionkey@0"
I20260808 04:04:03.079123 8349 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260808 04:04:03.082357 8370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:03.083595 8349 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260808 04:04:03.083770 8349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/minicluster-data/ts-2/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/minicluster-data/ts-2/wal
uuid: "09ea3124898843e0a7a68d99d83c9f83"
format_stamp: "Formatted at 2026-08-08 04:04:03 on dist-test-slave-1l3l"
server_key: "7c278e92cc735a9d3f3be1ce95d83c3c"
server_key_iv: "27a5bbac9f9064a6af3efb1a971da400"
server_key_version: "encryptionkey@0"
I20260808 04:04:03.083956 8349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:04:03.115365 8349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:04:03.119324 8349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:04:03.119580 8349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:04:03.122092 8349 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:04:03.122193 8349 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:03.122334 8349 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:04:03.122412 8349 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 04:04:03 dist-test-slave-1l3l krb5kdc[7810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161843, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.3@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:04:03.146891 8384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:03.123291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:45715 (local address 127.5.83.3:57633)
0808 04:04:03.124143 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:03.124193 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:03.126028 (+ 1835us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:03.126885 (+ 857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:03.126901 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:03.127736 (+ 835us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:04:03.128770 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:03.128803 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:03.130154 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:03.130159 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:03.132720 (+ 2561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:03.132739 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:03.133602 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:03.133610 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:03.133900 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:03.134697 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:04:03.134729 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:04:03.138308 (+ 3579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:04:03.142055 (+ 3747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:03.142064 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:03.142080 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:03.142545 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:03.143007 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:03.143012 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:03.143016 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:03.143294 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:03.143948 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:04:03.143956 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:04:03.144439 (+ 483us) client_negotiation.cc:769] Sending connection context
0808 04:04:03.144743 (+ 304us) client_negotiation.cc:240] Negotiation successful
0808 04:04:03.145322 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":389,"threads_started":1}
I20260808 04:04:03.148412 7938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:03.122761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.3:57633 (local address 127.5.83.62:45715)
0808 04:04:03.122951 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:03.122958 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:03.124318 (+ 1360us) server_negotiation.cc:407] Connection header received
0808 04:04:03.126294 (+ 1976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:03.126300 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:03.126391 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:03.126490 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:03.128986 (+ 2496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:03.129969 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:03.132887 (+ 2918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:03.133356 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:03.133464 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:03.138556 (+ 5092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:03.138562 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:03.138567 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:03.138613 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:03.141927 (+ 3314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:03.142678 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:03.142684 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:03.142688 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:03.142852 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:03.143408 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:03.143413 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:03.143417 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:03.143773 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:03.147735 (+ 3962us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:03.147860 (+ 125us) server_negotiation.cc:299] Negotiation successful
0808 04:04:03.148124 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 04:04:03.162050 8349 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.3:35133
I20260808 04:04:03.162079 8535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.3:35133 every 8 connection(s)
I20260808 04:04:03.163353 8349 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
I20260808 04:04:03.165225 8537 heartbeater.cc:347] Connected to a master server at 127.5.83.62:45715
I20260808 04:04:03.165444 8537 heartbeater.cc:464] Registering TS with master...
I20260808 04:04:03.166055 8537 heartbeater.cc:511] Master 127.5.83.62:45715 requested a full tablet report, sending...
I20260808 04:04:03.167378 7875 ts_manager.cc:194] Registered new tserver with Master: 09ea3124898843e0a7a68d99d83c9f83 (127.5.83.3:35133)
I20260808 04:04:03.167799 8513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:03.154524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:45715 (local address 127.5.83.3:54561)
0808 04:04:03.155157 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:03.155181 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:03.155331 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:03.155644 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:03.155647 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:03.156242 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:04:03.156750 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:03.156768 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:03.158005 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:03.158014 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:03.159394 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:03.159408 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:03.160245 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:03.160252 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:03.160429 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:03.161005 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:04:03.161030 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:04:03.162270 (+ 1240us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:04:03.165361 (+ 3091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:03.165366 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:03.165371 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:03.165757 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:03.166212 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:03.166217 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:03.166221 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:03.166339 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:03.166882 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:04:03.166888 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:04:03.167070 (+ 182us) client_negotiation.cc:769] Sending connection context
0808 04:04:03.167193 (+ 123us) client_negotiation.cc:240] Negotiation successful
0808 04:04:03.167406 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":565,"threads_started":1}
I20260808 04:04:03.167889 7938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:03.154468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.3:54561 (local address 127.5.83.62:45715)
0808 04:04:03.154645 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:03.154650 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:03.155264 (+ 614us) server_negotiation.cc:407] Connection header received
0808 04:04:03.155446 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:03.155451 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:03.155529 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:03.155632 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:03.156904 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:03.157883 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:03.159532 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:03.160016 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:03.160138 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:03.162439 (+ 2301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:03.162444 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:03.162447 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:03.162474 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:03.165250 (+ 2776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:03.165866 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:03.165871 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:03.165875 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:03.166097 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:03.166443 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:03.166447 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:03.166449 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:03.166759 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:03.166911 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:03.167308 (+ 397us) server_negotiation.cc:299] Negotiation successful
0808 04:04:03.167523 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 04:04:03.168133 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 8349
I20260808 04:04:03.168304 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/minicluster-data/ts-2/wal/instance
I20260808 04:04:03.168421 7875 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.3@KRBTEST.COM'} at 127.5.83.3:57633
I20260808 04:04:03.168706 5452 external_mini_cluster.cc:1428] Setting key 560da4b8e65970b71511cbe4bff21616
I20260808 04:04:03.171691 5452 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:04:03.179675 8159 heartbeater.cc:503] Master 127.5.83.62:45715 was elected leader, sending a full tablet report...
Aug 08 04:04:03 dist-test-slave-1l3l krb5kdc[7810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 04:04:03 dist-test-slave-1l3l krb5kdc[7810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:04:03.218863 7938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:03.206006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48898 (local address 127.5.83.62:45715)
0808 04:04:03.206235 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:03.206242 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:03.206515 (+ 273us) server_negotiation.cc:407] Connection header received
0808 04:04:03.206699 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:03.206703 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:03.206779 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:03.206867 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:03.208387 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:03.209105 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:03.210299 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:03.210727 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:03.210821 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:03.214417 (+ 3596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:03.214421 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:03.214425 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:03.214456 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:03.216827 (+ 2371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:03.217369 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:03.217372 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:03.217375 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:03.217500 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:03.217780 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:03.217783 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:03.217785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:03.218036 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:03.218223 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:03.218474 (+ 251us) server_negotiation.cc:299] Negotiation successful
0808 04:04:03.218638 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260808 04:04:03.222703 7874 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48898:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:04:03.229460 7874 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:04:03.262264 8557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:03.242740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:42739 (local address 127.0.0.1:56412)
0808 04:04:03.253331 (+ 10591us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:03.253350 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:03.253465 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:03.254633 (+ 1168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:03.254638 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:03.254666 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:04:03.255084 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:03.255096 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:03.257302 (+ 2206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:03.257307 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:03.259829 (+ 2522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:03.259844 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:03.261241 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:03.261247 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:03.261451 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:03.261495 (+ 44us) client_negotiation.cc:769] Sending connection context
0808 04:04:03.261601 (+ 106us) client_negotiation.cc:240] Negotiation successful
0808 04:04:03.261792 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10456,"mutex_wait_us":46,"thread_start_us":10801,"threads_started":1}
I20260808 04:04:03.262941 8552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:03.240796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.3:35133 (local address 127.0.0.1:41000)
0808 04:04:03.241684 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:03.241729 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:03.241920 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:03.254253 (+ 12333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:03.254261 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:03.254290 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:04:03.254693 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:03.254710 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:03.257698 (+ 2988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:03.257703 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:03.260029 (+ 2326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:03.260045 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:03.261855 (+ 1810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:03.261862 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:03.262035 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:03.262070 (+ 35us) client_negotiation.cc:769] Sending connection context
0808 04:04:03.262136 (+ 66us) client_negotiation.cc:240] Negotiation successful
0808 04:04:03.262267 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"mutex_wait_us":24,"thread_start_us":675,"threads_started":1}
I20260808 04:04:03.263145 8553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:03.241157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41000 (local address 127.5.83.3:35133)
0808 04:04:03.253606 (+ 12449us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:03.253615 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:03.253652 (+ 37us) server_negotiation.cc:407] Connection header received
0808 04:04:03.253757 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:03.253767 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:03.254028 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:03.254235 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:04:03.254983 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:03.257553 (+ 2570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:03.260231 (+ 2678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:03.261631 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:03.262380 (+ 749us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:03.262594 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:03.262669 (+ 75us) server_negotiation.cc:299] Negotiation successful
0808 04:04:03.262812 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12242,"thread_start_us":13326,"threads_started":1}
I20260808 04:04:03.263899 8554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:03.242152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.2:41193 (local address 127.0.0.1:57554)
0808 04:04:03.242706 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:03.242722 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:03.242826 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:03.255446 (+ 12620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:03.255451 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:03.255473 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:04:03.255820 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:03.255830 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:03.258862 (+ 3032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:03.258868 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:03.261238 (+ 2370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:03.261253 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:03.263214 (+ 1961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:03.263220 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:03.263393 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:03.263417 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 04:04:03.263494 (+ 77us) client_negotiation.cc:240] Negotiation successful
0808 04:04:03.263577 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"mutex_wait_us":132,"thread_start_us":272,"threads_started":1}
I20260808 04:04:03.264021 8558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:03.243202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56412 (local address 127.5.83.1:42739)
0808 04:04:03.254077 (+ 10875us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:03.254085 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:03.254107 (+ 22us) server_negotiation.cc:407] Connection header received
0808 04:04:03.254185 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:03.254191 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:03.254410 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:03.255332 (+ 922us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:04:03.255537 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:03.257154 (+ 1617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:03.260050 (+ 2896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:03.261047 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:03.261167 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:03.261378 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:03.261659 (+ 281us) server_negotiation.cc:299] Negotiation successful
0808 04:04:03.261825 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10739,"thread_start_us":10600,"threads_started":1}
I20260808 04:04:03.264284 8556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:03.242430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57554 (local address 127.5.83.2:41193)
0808 04:04:03.255029 (+ 12599us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:03.255035 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:03.255055 (+ 20us) server_negotiation.cc:407] Connection header received
0808 04:04:03.255124 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:03.255128 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:03.255264 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:03.255447 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:04:03.255992 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:03.258704 (+ 2712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:03.261443 (+ 2739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:03.262761 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:03.262885 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:03.263081 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:03.263579 (+ 498us) server_negotiation.cc:299] Negotiation successful
0808 04:04:03.263777 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12451,"thread_start_us":12336,"threads_started":1}
I20260808 04:04:03.266876 8057 tablet_service.cc:1467] Processing CreateTablet for tablet a637aee39ae44ca9a7d320f2c2739fdf (DEFAULT_TABLE table=test-table [id=ccb1901d4a944307a75aae933e2d1aaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:04:03.267014 8436 tablet_service.cc:1467] Processing CreateTablet for tablet a637aee39ae44ca9a7d320f2c2739fdf (DEFAULT_TABLE table=test-table [id=ccb1901d4a944307a75aae933e2d1aaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:04:03.267297 8249 tablet_service.cc:1467] Processing CreateTablet for tablet a637aee39ae44ca9a7d320f2c2739fdf (DEFAULT_TABLE table=test-table [id=ccb1901d4a944307a75aae933e2d1aaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:04:03.268142 8057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a637aee39ae44ca9a7d320f2c2739fdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:04:03.268432 8249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a637aee39ae44ca9a7d320f2c2739fdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:04:03.268441 8436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a637aee39ae44ca9a7d320f2c2739fdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:04:03.279768 8559 tablet_bootstrap.cc:492] T a637aee39ae44ca9a7d320f2c2739fdf P ac14eb089ed14bdbbd682f20cb7e9ada: Bootstrap starting.
I20260808 04:04:03.282213 8561 tablet_bootstrap.cc:492] T a637aee39ae44ca9a7d320f2c2739fdf P 09ea3124898843e0a7a68d99d83c9f83: Bootstrap starting.
I20260808 04:04:03.283021 8559 tablet_bootstrap.cc:654] T a637aee39ae44ca9a7d320f2c2739fdf P ac14eb089ed14bdbbd682f20cb7e9ada: Neither blocks nor log segments found. Creating new log.
I20260808 04:04:03.283552 8562 tablet_bootstrap.cc:492] T a637aee39ae44ca9a7d320f2c2739fdf P 1f623bcbb73947e8ba2becbff3aa228f: Bootstrap starting.
I20260808 04:04:03.284413 8559 log.cc:824] T a637aee39ae44ca9a7d320f2c2739fdf P ac14eb089ed14bdbbd682f20cb7e9ada: Log is configured to *not* fsync() on all Append() calls
I20260808 04:04:03.285461 8561 tablet_bootstrap.cc:654] T a637aee39ae44ca9a7d320f2c2739fdf P 09ea3124898843e0a7a68d99d83c9f83: Neither blocks nor log segments found. Creating new log.
I20260808 04:04:03.286402 8562 tablet_bootstrap.cc:654] T a637aee39ae44ca9a7d320f2c2739fdf P 1f623bcbb73947e8ba2becbff3aa228f: Neither blocks nor log segments found. Creating new log.
I20260808 04:04:03.286607 8561 log.cc:824] T a637aee39ae44ca9a7d320f2c2739fdf P 09ea3124898843e0a7a68d99d83c9f83: Log is configured to *not* fsync() on all Append() calls
I20260808 04:04:03.287410 8562 log.cc:824] T a637aee39ae44ca9a7d320f2c2739fdf P 1f623bcbb73947e8ba2becbff3aa228f: Log is configured to *not* fsync() on all Append() calls
I20260808 04:04:03.287519 8559 tablet_bootstrap.cc:492] T a637aee39ae44ca9a7d320f2c2739fdf P ac14eb089ed14bdbbd682f20cb7e9ada: No bootstrap required, opened a new log
I20260808 04:04:03.287817 8559 ts_tablet_manager.cc:1397] T a637aee39ae44ca9a7d320f2c2739fdf P ac14eb089ed14bdbbd682f20cb7e9ada: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.003s
I20260808 04:04:03.290258 8562 tablet_bootstrap.cc:492] T a637aee39ae44ca9a7d320f2c2739fdf P 1f623bcbb73947e8ba2becbff3aa228f: No bootstrap required, opened a new log
I20260808 04:04:03.290499 8562 ts_tablet_manager.cc:1397] T a637aee39ae44ca9a7d320f2c2739fdf P 1f623bcbb73947e8ba2becbff3aa228f: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260808 04:04:03.294199 8561 tablet_bootstrap.cc:492] T a637aee39ae44ca9a7d320f2c2739fdf P 09ea3124898843e0a7a68d99d83c9f83: No bootstrap required, opened a new log
I20260808 04:04:03.294456 8561 ts_tablet_manager.cc:1397] T a637aee39ae44ca9a7d320f2c2739fdf P 09ea3124898843e0a7a68d99d83c9f83: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260808 04:04:03.297562 8562 raft_consensus.cc:348] T a637aee39ae44ca9a7d320f2c2739fdf P 1f623bcbb73947e8ba2becbff3aa228f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ea3124898843e0a7a68d99d83c9f83" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 35133 } } peers { permanent_uuid: "1f623bcbb73947e8ba2becbff3aa228f" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 41193 } } peers { permanent_uuid: "ac14eb089ed14bdbbd682f20cb7e9ada" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42739 } }
I20260808 04:04:03.297890 8562 raft_consensus.cc:374] T a637aee39ae44ca9a7d320f2c2739fdf P 1f623bcbb73947e8ba2becbff3aa228f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:04:03.297981 8562 raft_consensus.cc:729] T a637aee39ae44ca9a7d320f2c2739fdf P 1f623bcbb73947e8ba2becbff3aa228f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f623bcbb73947e8ba2becbff3aa228f, State: Initialized, Role: FOLLOWER
I20260808 04:04:03.298306 8559 raft_consensus.cc:348] T a637aee39ae44ca9a7d320f2c2739fdf P ac14eb089ed14bdbbd682f20cb7e9ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ea3124898843e0a7a68d99d83c9f83" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 35133 } } peers { permanent_uuid: "1f623bcbb73947e8ba2becbff3aa228f" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 41193 } } peers { permanent_uuid: "ac14eb089ed14bdbbd682f20cb7e9ada" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42739 } }
I20260808 04:04:03.298707 8559 raft_consensus.cc:374] T a637aee39ae44ca9a7d320f2c2739fdf P ac14eb089ed14bdbbd682f20cb7e9ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:04:03.298573 8562 consensus_queue.cc:260] T a637aee39ae44ca9a7d320f2c2739fdf P 1f623bcbb73947e8ba2becbff3aa228f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ea3124898843e0a7a68d99d83c9f83" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 35133 } } peers { permanent_uuid: "1f623bcbb73947e8ba2becbff3aa228f" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 41193 } } peers { permanent_uuid: "ac14eb089ed14bdbbd682f20cb7e9ada" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42739 } }
I20260808 04:04:03.298806 8559 raft_consensus.cc:729] T a637aee39ae44ca9a7d320f2c2739fdf P ac14eb089ed14bdbbd682f20cb7e9ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac14eb089ed14bdbbd682f20cb7e9ada, State: Initialized, Role: FOLLOWER
I20260808 04:04:03.299342 8559 consensus_queue.cc:260] T a637aee39ae44ca9a7d320f2c2739fdf P ac14eb089ed14bdbbd682f20cb7e9ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ea3124898843e0a7a68d99d83c9f83" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 35133 } } peers { permanent_uuid: "1f623bcbb73947e8ba2becbff3aa228f" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 41193 } } peers { permanent_uuid: "ac14eb089ed14bdbbd682f20cb7e9ada" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42739 } }
I20260808 04:04:03.300985 8559 ts_tablet_manager.cc:1428] T a637aee39ae44ca9a7d320f2c2739fdf P ac14eb089ed14bdbbd682f20cb7e9ada: Time spent starting tablet: real 0.013s user 0.011s sys 0.001s
I20260808 04:04:03.301812 8345 heartbeater.cc:503] Master 127.5.83.62:45715 was elected leader, sending a full tablet report...
I20260808 04:04:03.302637 8562 ts_tablet_manager.cc:1428] T a637aee39ae44ca9a7d320f2c2739fdf P 1f623bcbb73947e8ba2becbff3aa228f: Time spent starting tablet: real 0.012s user 0.009s sys 0.004s
I20260808 04:04:03.304237 8561 raft_consensus.cc:348] T a637aee39ae44ca9a7d320f2c2739fdf P 09ea3124898843e0a7a68d99d83c9f83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ea3124898843e0a7a68d99d83c9f83" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 35133 } } peers { permanent_uuid: "1f623bcbb73947e8ba2becbff3aa228f" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 41193 } } peers { permanent_uuid: "ac14eb089ed14bdbbd682f20cb7e9ada" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42739 } }
I20260808 04:04:03.304567 8561 raft_consensus.cc:374] T a637aee39ae44ca9a7d320f2c2739fdf P 09ea3124898843e0a7a68d99d83c9f83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:04:03.304672 8561 raft_consensus.cc:729] T a637aee39ae44ca9a7d320f2c2739fdf P 09ea3124898843e0a7a68d99d83c9f83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ea3124898843e0a7a68d99d83c9f83, State: Initialized, Role: FOLLOWER
I20260808 04:04:03.305212 8561 consensus_queue.cc:260] T a637aee39ae44ca9a7d320f2c2739fdf P 09ea3124898843e0a7a68d99d83c9f83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ea3124898843e0a7a68d99d83c9f83" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 35133 } } peers { permanent_uuid: "1f623bcbb73947e8ba2becbff3aa228f" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 41193 } } peers { permanent_uuid: "ac14eb089ed14bdbbd682f20cb7e9ada" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42739 } }
I20260808 04:04:03.306249 8537 heartbeater.cc:503] Master 127.5.83.62:45715 was elected leader, sending a full tablet report...
I20260808 04:04:03.306870 8561 ts_tablet_manager.cc:1428] T a637aee39ae44ca9a7d320f2c2739fdf P 09ea3124898843e0a7a68d99d83c9f83: Time spent starting tablet: real 0.012s user 0.009s sys 0.004s
W20260808 04:04:03.414304 8346 tablet.cc:2369] T a637aee39ae44ca9a7d320f2c2739fdf P 1f623bcbb73947e8ba2becbff3aa228f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 04:04:03.414954 8538 tablet.cc:2369] T a637aee39ae44ca9a7d320f2c2739fdf P 09ea3124898843e0a7a68d99d83c9f83: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 04:04:03.423102 8161 tablet.cc:2369] T a637aee39ae44ca9a7d320f2c2739fdf P ac14eb089ed14bdbbd682f20cb7e9ada: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:04:03.529223 8570 raft_consensus.cc:482] T a637aee39ae44ca9a7d320f2c2739fdf P 1f623bcbb73947e8ba2becbff3aa228f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:04:03.529561 8570 raft_consensus.cc:504] T a637aee39ae44ca9a7d320f2c2739fdf P 1f623bcbb73947e8ba2becbff3aa228f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ea3124898843e0a7a68d99d83c9f83" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 35133 } } peers { permanent_uuid: "1f623bcbb73947e8ba2becbff3aa228f" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 41193 } } peers { permanent_uuid: "ac14eb089ed14bdbbd682f20cb7e9ada" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42739 } }
I20260808 04:04:03.531200 8570 leader_election.cc:288] T a637aee39ae44ca9a7d320f2c2739fdf P 1f623bcbb73947e8ba2becbff3aa228f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ea3124898843e0a7a68d99d83c9f83 (127.5.83.3:35133), ac14eb089ed14bdbbd682f20cb7e9ada (127.5.83.1:42739)
I20260808 04:04:03.538877 8575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:03.531667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.3:35133 (local address 127.5.83.2:38083)
0808 04:04:03.532486 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:03.532507 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:03.532712 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:03.533277 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:03.533282 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:03.533315 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:04:03.533827 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:03.533839 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:03.535126 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:03.535132 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:03.536829 (+ 1697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:03.536842 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:03.538156 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:03.538163 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:03.538328 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:03.538362 (+ 34us) client_negotiation.cc:769] Sending connection context
0808 04:04:03.538462 (+ 100us) client_negotiation.cc:240] Negotiation successful
0808 04:04:03.538595 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"thread_start_us":398,"threads_started":1}
I20260808 04:04:03.538902 8553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:03.531837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.2:38083 (local address 127.5.83.3:35133)
0808 04:04:03.532042 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:03.532049 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:03.532585 (+ 536us) server_negotiation.cc:407] Connection header received
0808 04:04:03.532886 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:03.532892 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:03.532974 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:03.533048 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:04:03.533988 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:03.535001 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:03.537041 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:03.537939 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:03.538078 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:03.538171 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:03.538509 (+ 338us) server_negotiation.cc:299] Negotiation successful
0808 04:04:03.538636 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 04:04:03.539295 8558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:03.532828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.2:46497 (local address 127.5.83.1:42739)
0808 04:04:03.533050 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:03.533055 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:03.533915 (+ 860us) server_negotiation.cc:407] Connection header received
0808 04:04:03.534043 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:03.534046 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:03.534120 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:03.534186 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:04:03.534741 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:03.535629 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:03.537437 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:03.538347 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:03.538481 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:03.538576 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:03.538978 (+ 402us) server_negotiation.cc:299] Negotiation successful
0808 04:04:03.539096 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260808 04:04:03.539547 8576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:03.532580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:42739 (local address 127.5.83.2:46497)
0808 04:04:03.533784 (+ 1204us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:03.533814 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:03.533930 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:03.534226 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:03.534231 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:03.534259 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:04:03.534609 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:03.534621 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:03.535789 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:03.535793 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:03.537288 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:03.537304 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:03.538584 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:03.538591 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:03.538766 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:03.538826 (+ 60us) client_negotiation.cc:769] Sending connection context
0808 04:04:03.539064 (+ 238us) client_negotiation.cc:240] Negotiation successful
0808 04:04:03.539187 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1081,"mutex_wait_us":5,"thread_start_us":867,"threads_started":1}
I20260808 04:04:03.540189 8461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a637aee39ae44ca9a7d320f2c2739fdf" candidate_uuid: "1f623bcbb73947e8ba2becbff3aa228f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ea3124898843e0a7a68d99d83c9f83" is_pre_election: true
I20260808 04:04:03.540553 8091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a637aee39ae44ca9a7d320f2c2739fdf" candidate_uuid: "1f623bcbb73947e8ba2becbff3aa228f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac14eb089ed14bdbbd682f20cb7e9ada" is_pre_election: true
I20260808 04:04:03.540709 8461 raft_consensus.cc:2454] T a637aee39ae44ca9a7d320f2c2739fdf P 09ea3124898843e0a7a68d99d83c9f83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f623bcbb73947e8ba2becbff3aa228f in term 0.
I20260808 04:04:03.541039 8091 raft_consensus.cc:2454] T a637aee39ae44ca9a7d320f2c2739fdf P ac14eb089ed14bdbbd682f20cb7e9ada [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f623bcbb73947e8ba2becbff3aa228f in term 0.
I20260808 04:04:03.541390 8187 leader_election.cc:302] T a637aee39ae44ca9a7d320f2c2739fdf P 1f623bcbb73947e8ba2becbff3aa228f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ea3124898843e0a7a68d99d83c9f83, 1f623bcbb73947e8ba2becbff3aa228f; no voters:
I20260808 04:04:03.541816 8570 raft_consensus.cc:2781] T a637aee39ae44ca9a7d320f2c2739fdf P 1f623bcbb73947e8ba2becbff3aa228f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:04:03.541975 8570 raft_consensus.cc:482] T a637aee39ae44ca9a7d320f2c2739fdf P 1f623bcbb73947e8ba2becbff3aa228f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:04:03.542085 8570 raft_consensus.cc:3037] T a637aee39ae44ca9a7d320f2c2739fdf P 1f623bcbb73947e8ba2becbff3aa228f [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:04:03.544438 8570 raft_consensus.cc:504] T a637aee39ae44ca9a7d320f2c2739fdf P 1f623bcbb73947e8ba2becbff3aa228f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ea3124898843e0a7a68d99d83c9f83" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 35133 } } peers { permanent_uuid: "1f623bcbb73947e8ba2becbff3aa228f" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 41193 } } peers { permanent_uuid: "ac14eb089ed14bdbbd682f20cb7e9ada" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42739 } }
I20260808 04:04:03.545013 8570 leader_election.cc:288] T a637aee39ae44ca9a7d320f2c2739fdf P 1f623bcbb73947e8ba2becbff3aa228f [CANDIDATE]: Term 1 election: Requested vote from peers 09ea3124898843e0a7a68d99d83c9f83 (127.5.83.3:35133), ac14eb089ed14bdbbd682f20cb7e9ada (127.5.83.1:42739)
I20260808 04:04:03.545425 8461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a637aee39ae44ca9a7d320f2c2739fdf" candidate_uuid: "1f623bcbb73947e8ba2becbff3aa228f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ea3124898843e0a7a68d99d83c9f83"
I20260808 04:04:03.545485 8091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a637aee39ae44ca9a7d320f2c2739fdf" candidate_uuid: "1f623bcbb73947e8ba2becbff3aa228f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac14eb089ed14bdbbd682f20cb7e9ada"
I20260808 04:04:03.545603 8461 raft_consensus.cc:3037] T a637aee39ae44ca9a7d320f2c2739fdf P 09ea3124898843e0a7a68d99d83c9f83 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:04:03.545658 8091 raft_consensus.cc:3037] T a637aee39ae44ca9a7d320f2c2739fdf P ac14eb089ed14bdbbd682f20cb7e9ada [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:04:03.547752 8091 raft_consensus.cc:2454] T a637aee39ae44ca9a7d320f2c2739fdf P ac14eb089ed14bdbbd682f20cb7e9ada [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f623bcbb73947e8ba2becbff3aa228f in term 1.
I20260808 04:04:03.547778 8461 raft_consensus.cc:2454] T a637aee39ae44ca9a7d320f2c2739fdf P 09ea3124898843e0a7a68d99d83c9f83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f623bcbb73947e8ba2becbff3aa228f in term 1.
I20260808 04:04:03.548110 8187 leader_election.cc:302] T a637aee39ae44ca9a7d320f2c2739fdf P 1f623bcbb73947e8ba2becbff3aa228f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ea3124898843e0a7a68d99d83c9f83, 1f623bcbb73947e8ba2becbff3aa228f; no voters:
I20260808 04:04:03.548357 8570 raft_consensus.cc:2781] T a637aee39ae44ca9a7d320f2c2739fdf P 1f623bcbb73947e8ba2becbff3aa228f [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:04:03.548767 8570 raft_consensus.cc:686] T a637aee39ae44ca9a7d320f2c2739fdf P 1f623bcbb73947e8ba2becbff3aa228f [term 1 LEADER]: Becoming Leader. State: Replica: 1f623bcbb73947e8ba2becbff3aa228f, State: Running, Role: LEADER
I20260808 04:04:03.549177 8570 consensus_queue.cc:237] T a637aee39ae44ca9a7d320f2c2739fdf P 1f623bcbb73947e8ba2becbff3aa228f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ea3124898843e0a7a68d99d83c9f83" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 35133 } } peers { permanent_uuid: "1f623bcbb73947e8ba2becbff3aa228f" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 41193 } } peers { permanent_uuid: "ac14eb089ed14bdbbd682f20cb7e9ada" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42739 } }
I20260808 04:04:03.553295 7873 catalog_manager.cc:5495] T a637aee39ae44ca9a7d320f2c2739fdf P 1f623bcbb73947e8ba2becbff3aa228f reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f623bcbb73947e8ba2becbff3aa228f (127.5.83.2). New cstate: current_term: 1 leader_uuid: "1f623bcbb73947e8ba2becbff3aa228f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ea3124898843e0a7a68d99d83c9f83" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 35133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f623bcbb73947e8ba2becbff3aa228f" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 41193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac14eb089ed14bdbbd682f20cb7e9ada" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42739 } health_report { overall_health: UNKNOWN } } }
Aug 08 04:04:03 dist-test-slave-1l3l krb5kdc[7810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161841, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.62@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:04:03.646854 8587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:03.634179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:45715 (local address 127.0.0.1:48908)
0808 04:04:03.634815 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:03.634832 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:03.634972 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:03.635260 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:03.635263 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:03.635985 (+ 722us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:04:03.636237 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:03.636246 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:03.637250 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:03.637254 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:03.638266 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:03.638278 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:03.638998 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:03.639003 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:03.639125 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:03.639773 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:04:03.639797 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:04:03.641910 (+ 2113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:04:03.644619 (+ 2709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:03.644629 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:03.644635 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:03.645110 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:03.645390 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:03.645393 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:03.645395 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:03.645515 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:03.646016 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:04:03.646022 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:04:03.646172 (+ 150us) client_negotiation.cc:769] Sending connection context
0808 04:04:03.646303 (+ 131us) client_negotiation.cc:240] Negotiation successful
0808 04:04:03.646499 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":360,"threads_started":1}
I20260808 04:04:03.646880 7938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:03.634113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48908 (local address 127.5.83.62:45715)
0808 04:04:03.634313 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:03.634319 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:03.634919 (+ 600us) server_negotiation.cc:407] Connection header received
0808 04:04:03.635064 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:03.635068 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:03.635140 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:03.635222 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:03.636399 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:03.637153 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:03.638426 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:03.638844 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:03.638941 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:03.642079 (+ 3138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:03.642083 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:03.642086 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:03.642114 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:03.644513 (+ 2399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:03.645202 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:03.645205 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:03.645208 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:03.645314 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:03.645596 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:03.645599 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:03.645601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:03.645913 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:03.646021 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:03.646379 (+ 358us) server_negotiation.cc:299] Negotiation successful
0808 04:04:03.646612 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":70}
I20260808 04:04:03.651499 7873 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.5.83.62@KRBTEST.COM'} at 127.0.0.1:48908:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260808 04:04:03.652537 7873 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:04:03.658783 8249 tablet_service.cc:1467] Processing CreateTablet for tablet f0d6fccc8fbb4cb1819c89f2296f8838 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fe7d17a6d8e04c53b5ab907df014197a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 04:04:03.658802 8436 tablet_service.cc:1467] Processing CreateTablet for tablet f0d6fccc8fbb4cb1819c89f2296f8838 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fe7d17a6d8e04c53b5ab907df014197a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 04:04:03.658691 8057 tablet_service.cc:1467] Processing CreateTablet for tablet f0d6fccc8fbb4cb1819c89f2296f8838 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fe7d17a6d8e04c53b5ab907df014197a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 04:04:03.659659 8249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0d6fccc8fbb4cb1819c89f2296f8838. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:04:03.659672 8436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0d6fccc8fbb4cb1819c89f2296f8838. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:04:03.659672 8057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0d6fccc8fbb4cb1819c89f2296f8838. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:04:03.666693 8559 tablet_bootstrap.cc:492] T f0d6fccc8fbb4cb1819c89f2296f8838 P ac14eb089ed14bdbbd682f20cb7e9ada: Bootstrap starting.
I20260808 04:04:03.666720 8562 tablet_bootstrap.cc:492] T f0d6fccc8fbb4cb1819c89f2296f8838 P 1f623bcbb73947e8ba2becbff3aa228f: Bootstrap starting.
I20260808 04:04:03.668296 8561 tablet_bootstrap.cc:492] T f0d6fccc8fbb4cb1819c89f2296f8838 P 09ea3124898843e0a7a68d99d83c9f83: Bootstrap starting.
I20260808 04:04:03.668349 8559 tablet_bootstrap.cc:654] T f0d6fccc8fbb4cb1819c89f2296f8838 P ac14eb089ed14bdbbd682f20cb7e9ada: Neither blocks nor log segments found. Creating new log.
I20260808 04:04:03.668350 8562 tablet_bootstrap.cc:654] T f0d6fccc8fbb4cb1819c89f2296f8838 P 1f623bcbb73947e8ba2becbff3aa228f: Neither blocks nor log segments found. Creating new log.
I20260808 04:04:03.669802 8561 tablet_bootstrap.cc:654] T f0d6fccc8fbb4cb1819c89f2296f8838 P 09ea3124898843e0a7a68d99d83c9f83: Neither blocks nor log segments found. Creating new log.
I20260808 04:04:03.678834 8561 tablet_bootstrap.cc:492] T f0d6fccc8fbb4cb1819c89f2296f8838 P 09ea3124898843e0a7a68d99d83c9f83: No bootstrap required, opened a new log
I20260808 04:04:03.678973 8561 ts_tablet_manager.cc:1397] T f0d6fccc8fbb4cb1819c89f2296f8838 P 09ea3124898843e0a7a68d99d83c9f83: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260808 04:04:03.679663 8561 raft_consensus.cc:348] T f0d6fccc8fbb4cb1819c89f2296f8838 P 09ea3124898843e0a7a68d99d83c9f83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac14eb089ed14bdbbd682f20cb7e9ada" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42739 } } peers { permanent_uuid: "1f623bcbb73947e8ba2becbff3aa228f" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 41193 } } peers { permanent_uuid: "09ea3124898843e0a7a68d99d83c9f83" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 35133 } }
I20260808 04:04:03.679943 8561 raft_consensus.cc:374] T f0d6fccc8fbb4cb1819c89f2296f8838 P 09ea3124898843e0a7a68d99d83c9f83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:04:03.680053 8562 tablet_bootstrap.cc:492] T f0d6fccc8fbb4cb1819c89f2296f8838 P 1f623bcbb73947e8ba2becbff3aa228f: No bootstrap required, opened a new log
I20260808 04:04:03.680050 8561 raft_consensus.cc:729] T f0d6fccc8fbb4cb1819c89f2296f8838 P 09ea3124898843e0a7a68d99d83c9f83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ea3124898843e0a7a68d99d83c9f83, State: Initialized, Role: FOLLOWER
I20260808 04:04:03.680204 8562 ts_tablet_manager.cc:1397] T f0d6fccc8fbb4cb1819c89f2296f8838 P 1f623bcbb73947e8ba2becbff3aa228f: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.007s
I20260808 04:04:03.680322 8559 tablet_bootstrap.cc:492] T f0d6fccc8fbb4cb1819c89f2296f8838 P ac14eb089ed14bdbbd682f20cb7e9ada: No bootstrap required, opened a new log
I20260808 04:04:03.680322 8561 consensus_queue.cc:260] T f0d6fccc8fbb4cb1819c89f2296f8838 P 09ea3124898843e0a7a68d99d83c9f83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac14eb089ed14bdbbd682f20cb7e9ada" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42739 } } peers { permanent_uuid: "1f623bcbb73947e8ba2becbff3aa228f" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 41193 } } peers { permanent_uuid: "09ea3124898843e0a7a68d99d83c9f83" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 35133 } }
I20260808 04:04:03.680497 8559 ts_tablet_manager.cc:1397] T f0d6fccc8fbb4cb1819c89f2296f8838 P ac14eb089ed14bdbbd682f20cb7e9ada: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.003s
I20260808 04:04:03.680828 8573 tablet_replica.cc:439] T f0d6fccc8fbb4cb1819c89f2296f8838 P 09ea3124898843e0a7a68d99d83c9f83: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac14eb089ed14bdbbd682f20cb7e9ada" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42739 } } peers { permanent_uuid: "1f623bcbb73947e8ba2becbff3aa228f" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 41193 } } peers { permanent_uuid: "09ea3124898843e0a7a68d99d83c9f83" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 35133 } } }
I20260808 04:04:03.681106 8573 tablet_replica.cc:442] T f0d6fccc8fbb4cb1819c89f2296f8838 P 09ea3124898843e0a7a68d99d83c9f83: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 04:04:03.681142 8562 raft_consensus.cc:348] T f0d6fccc8fbb4cb1819c89f2296f8838 P 1f623bcbb73947e8ba2becbff3aa228f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac14eb089ed14bdbbd682f20cb7e9ada" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42739 } } peers { permanent_uuid: "1f623bcbb73947e8ba2becbff3aa228f" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 41193 } } peers { permanent_uuid: "09ea3124898843e0a7a68d99d83c9f83" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 35133 } }
I20260808 04:04:03.681391 8562 raft_consensus.cc:374] T f0d6fccc8fbb4cb1819c89f2296f8838 P 1f623bcbb73947e8ba2becbff3aa228f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:04:03.681514 8562 raft_consensus.cc:729] T f0d6fccc8fbb4cb1819c89f2296f8838 P 1f623bcbb73947e8ba2becbff3aa228f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f623bcbb73947e8ba2becbff3aa228f, State: Initialized, Role: FOLLOWER
I20260808 04:04:03.681730 8559 raft_consensus.cc:348] T f0d6fccc8fbb4cb1819c89f2296f8838 P ac14eb089ed14bdbbd682f20cb7e9ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac14eb089ed14bdbbd682f20cb7e9ada" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42739 } } peers { permanent_uuid: "1f623bcbb73947e8ba2becbff3aa228f" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 41193 } } peers { permanent_uuid: "09ea3124898843e0a7a68d99d83c9f83" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 35133 } }
I20260808 04:04:03.681779 8562 consensus_queue.cc:260] T f0d6fccc8fbb4cb1819c89f2296f8838 P 1f623bcbb73947e8ba2becbff3aa228f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac14eb089ed14bdbbd682f20cb7e9ada" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42739 } } peers { permanent_uuid: "1f623bcbb73947e8ba2becbff3aa228f" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 41193 } } peers { permanent_uuid: "09ea3124898843e0a7a68d99d83c9f83" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 35133 } }
I20260808 04:04:03.682046 8559 raft_consensus.cc:374] T f0d6fccc8fbb4cb1819c89f2296f8838 P ac14eb089ed14bdbbd682f20cb7e9ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:04:03.682197 8559 raft_consensus.cc:729] T f0d6fccc8fbb4cb1819c89f2296f8838 P ac14eb089ed14bdbbd682f20cb7e9ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac14eb089ed14bdbbd682f20cb7e9ada, State: Initialized, Role: FOLLOWER
I20260808 04:04:03.682370 8562 ts_tablet_manager.cc:1428] T f0d6fccc8fbb4cb1819c89f2296f8838 P 1f623bcbb73947e8ba2becbff3aa228f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 04:04:03.682274 8570 tablet_replica.cc:439] T f0d6fccc8fbb4cb1819c89f2296f8838 P 1f623bcbb73947e8ba2becbff3aa228f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac14eb089ed14bdbbd682f20cb7e9ada" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42739 } } peers { permanent_uuid: "1f623bcbb73947e8ba2becbff3aa228f" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 41193 } } peers { permanent_uuid: "09ea3124898843e0a7a68d99d83c9f83" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 35133 } } }
I20260808 04:04:03.682565 8570 tablet_replica.cc:442] T f0d6fccc8fbb4cb1819c89f2296f8838 P 1f623bcbb73947e8ba2becbff3aa228f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 04:04:03.682503 8559 consensus_queue.cc:260] T f0d6fccc8fbb4cb1819c89f2296f8838 P ac14eb089ed14bdbbd682f20cb7e9ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac14eb089ed14bdbbd682f20cb7e9ada" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42739 } } peers { permanent_uuid: "1f623bcbb73947e8ba2becbff3aa228f" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 41193 } } peers { permanent_uuid: "09ea3124898843e0a7a68d99d83c9f83" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 35133 } }
I20260808 04:04:03.683157 8559 ts_tablet_manager.cc:1428] T f0d6fccc8fbb4cb1819c89f2296f8838 P ac14eb089ed14bdbbd682f20cb7e9ada: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 04:04:03.682998 8572 tablet_replica.cc:439] T f0d6fccc8fbb4cb1819c89f2296f8838 P ac14eb089ed14bdbbd682f20cb7e9ada: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac14eb089ed14bdbbd682f20cb7e9ada" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42739 } } peers { permanent_uuid: "1f623bcbb73947e8ba2becbff3aa228f" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 41193 } } peers { permanent_uuid: "09ea3124898843e0a7a68d99d83c9f83" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 35133 } } }
I20260808 04:04:03.683363 8572 tablet_replica.cc:442] T f0d6fccc8fbb4cb1819c89f2296f8838 P ac14eb089ed14bdbbd682f20cb7e9ada: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 04:04:03.684072 8561 ts_tablet_manager.cc:1428] T f0d6fccc8fbb4cb1819c89f2296f8838 P 09ea3124898843e0a7a68d99d83c9f83: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260808 04:04:03.931448 8572 raft_consensus.cc:482] T f0d6fccc8fbb4cb1819c89f2296f8838 P ac14eb089ed14bdbbd682f20cb7e9ada [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:04:03.931648 8572 raft_consensus.cc:504] T f0d6fccc8fbb4cb1819c89f2296f8838 P ac14eb089ed14bdbbd682f20cb7e9ada [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac14eb089ed14bdbbd682f20cb7e9ada" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42739 } } peers { permanent_uuid: "1f623bcbb73947e8ba2becbff3aa228f" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 41193 } } peers { permanent_uuid: "09ea3124898843e0a7a68d99d83c9f83" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 35133 } }
I20260808 04:04:03.933185 8572 leader_election.cc:288] T f0d6fccc8fbb4cb1819c89f2296f8838 P ac14eb089ed14bdbbd682f20cb7e9ada [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f623bcbb73947e8ba2becbff3aa228f (127.5.83.2:41193), 09ea3124898843e0a7a68d99d83c9f83 (127.5.83.3:35133)
I20260808 04:04:03.939798 8592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:03.933547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.1:35633 (local address 127.5.83.2:41193)
0808 04:04:03.934611 (+ 1064us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:03.934617 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:03.934635 (+ 18us) server_negotiation.cc:407] Connection header received
0808 04:04:03.934698 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:03.934701 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:03.934757 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:03.934844 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:04:03.935452 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:03.936262 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:03.938028 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:03.938831 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:03.938962 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:03.939065 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:03.939331 (+ 266us) server_negotiation.cc:299] Negotiation successful
0808 04:04:03.939457 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":959,"thread_start_us":849,"threads_started":1}
I20260808 04:04:03.939862 8593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:03.933546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.2:41193 (local address 127.5.83.1:35633)
0808 04:04:03.934074 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:03.934090 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:03.934218 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:03.934885 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:03.934890 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:03.934923 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:04:03.935285 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:03.935294 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:03.936392 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:03.936395 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:03.937875 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:03.937897 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:03.939041 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:03.939048 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:03.939183 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:03.939207 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 04:04:03.939295 (+ 88us) client_negotiation.cc:240] Negotiation successful
0808 04:04:03.939432 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"spinlock_wait_cycles":61312,"thread_start_us":254,"threads_started":1}
I20260808 04:04:03.940052 8553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:03.934421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.1:43181 (local address 127.5.83.3:35133)
0808 04:04:03.934598 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:03.934603 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:03.935040 (+ 437us) server_negotiation.cc:407] Connection header received
0808 04:04:03.935246 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:03.935250 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:03.935323 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:03.935396 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:04:03.935950 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:03.936653 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:03.938404 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:03.939127 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:03.939247 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:03.939341 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:03.939647 (+ 306us) server_negotiation.cc:299] Negotiation successful
0808 04:04:03.939790 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 04:04:03.940363 8594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:03.934293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.3:35133 (local address 127.5.83.1:43181)
0808 04:04:03.934927 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:03.934942 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:03.935107 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:03.935440 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:03.935445 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:03.935468 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:04:03.935786 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:03.935795 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:03.936788 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:03.936793 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:03.938257 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:03.938269 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:03.939342 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:03.939349 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:03.939500 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:03.939522 (+ 22us) client_negotiation.cc:769] Sending connection context
0808 04:04:03.939603 (+ 81us) client_negotiation.cc:240] Negotiation successful
0808 04:04:03.939748 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":335,"threads_started":1}
I20260808 04:04:03.940943 8461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0d6fccc8fbb4cb1819c89f2296f8838" candidate_uuid: "ac14eb089ed14bdbbd682f20cb7e9ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ea3124898843e0a7a68d99d83c9f83" is_pre_election: true
I20260808 04:04:03.941028 8283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0d6fccc8fbb4cb1819c89f2296f8838" candidate_uuid: "ac14eb089ed14bdbbd682f20cb7e9ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f623bcbb73947e8ba2becbff3aa228f" is_pre_election: true
I20260808 04:04:03.941179 8461 raft_consensus.cc:2454] T f0d6fccc8fbb4cb1819c89f2296f8838 P 09ea3124898843e0a7a68d99d83c9f83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac14eb089ed14bdbbd682f20cb7e9ada in term 0.
I20260808 04:04:03.941407 8283 raft_consensus.cc:2454] T f0d6fccc8fbb4cb1819c89f2296f8838 P 1f623bcbb73947e8ba2becbff3aa228f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac14eb089ed14bdbbd682f20cb7e9ada in term 0.
I20260808 04:04:03.941560 7992 leader_election.cc:302] T f0d6fccc8fbb4cb1819c89f2296f8838 P ac14eb089ed14bdbbd682f20cb7e9ada [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ea3124898843e0a7a68d99d83c9f83, ac14eb089ed14bdbbd682f20cb7e9ada; no voters:
I20260808 04:04:03.941946 8572 raft_consensus.cc:2781] T f0d6fccc8fbb4cb1819c89f2296f8838 P ac14eb089ed14bdbbd682f20cb7e9ada [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:04:03.942063 8572 raft_consensus.cc:482] T f0d6fccc8fbb4cb1819c89f2296f8838 P ac14eb089ed14bdbbd682f20cb7e9ada [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:04:03.942167 8572 raft_consensus.cc:3037] T f0d6fccc8fbb4cb1819c89f2296f8838 P ac14eb089ed14bdbbd682f20cb7e9ada [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:04:03.944535 8572 raft_consensus.cc:504] T f0d6fccc8fbb4cb1819c89f2296f8838 P ac14eb089ed14bdbbd682f20cb7e9ada [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac14eb089ed14bdbbd682f20cb7e9ada" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42739 } } peers { permanent_uuid: "1f623bcbb73947e8ba2becbff3aa228f" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 41193 } } peers { permanent_uuid: "09ea3124898843e0a7a68d99d83c9f83" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 35133 } }
I20260808 04:04:03.945016 8572 leader_election.cc:288] T f0d6fccc8fbb4cb1819c89f2296f8838 P ac14eb089ed14bdbbd682f20cb7e9ada [CANDIDATE]: Term 1 election: Requested vote from peers 1f623bcbb73947e8ba2becbff3aa228f (127.5.83.2:41193), 09ea3124898843e0a7a68d99d83c9f83 (127.5.83.3:35133)
I20260808 04:04:03.945436 8283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0d6fccc8fbb4cb1819c89f2296f8838" candidate_uuid: "ac14eb089ed14bdbbd682f20cb7e9ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f623bcbb73947e8ba2becbff3aa228f"
I20260808 04:04:03.945497 8461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0d6fccc8fbb4cb1819c89f2296f8838" candidate_uuid: "ac14eb089ed14bdbbd682f20cb7e9ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ea3124898843e0a7a68d99d83c9f83"
I20260808 04:04:03.945602 8283 raft_consensus.cc:3037] T f0d6fccc8fbb4cb1819c89f2296f8838 P 1f623bcbb73947e8ba2becbff3aa228f [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:04:03.945675 8461 raft_consensus.cc:3037] T f0d6fccc8fbb4cb1819c89f2296f8838 P 09ea3124898843e0a7a68d99d83c9f83 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:04:03.947546 8461 raft_consensus.cc:2454] T f0d6fccc8fbb4cb1819c89f2296f8838 P 09ea3124898843e0a7a68d99d83c9f83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac14eb089ed14bdbbd682f20cb7e9ada in term 1.
I20260808 04:04:03.947578 8283 raft_consensus.cc:2454] T f0d6fccc8fbb4cb1819c89f2296f8838 P 1f623bcbb73947e8ba2becbff3aa228f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac14eb089ed14bdbbd682f20cb7e9ada in term 1.
I20260808 04:04:03.947921 7992 leader_election.cc:302] T f0d6fccc8fbb4cb1819c89f2296f8838 P ac14eb089ed14bdbbd682f20cb7e9ada [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ea3124898843e0a7a68d99d83c9f83, ac14eb089ed14bdbbd682f20cb7e9ada; no voters:
I20260808 04:04:03.948179 8572 raft_consensus.cc:2781] T f0d6fccc8fbb4cb1819c89f2296f8838 P ac14eb089ed14bdbbd682f20cb7e9ada [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:04:03.948556 8572 raft_consensus.cc:686] T f0d6fccc8fbb4cb1819c89f2296f8838 P ac14eb089ed14bdbbd682f20cb7e9ada [term 1 LEADER]: Becoming Leader. State: Replica: ac14eb089ed14bdbbd682f20cb7e9ada, State: Running, Role: LEADER
I20260808 04:04:03.948982 8572 consensus_queue.cc:237] T f0d6fccc8fbb4cb1819c89f2296f8838 P ac14eb089ed14bdbbd682f20cb7e9ada [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac14eb089ed14bdbbd682f20cb7e9ada" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42739 } } peers { permanent_uuid: "1f623bcbb73947e8ba2becbff3aa228f" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 41193 } } peers { permanent_uuid: "09ea3124898843e0a7a68d99d83c9f83" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 35133 } }
I20260808 04:04:03.952117 8596 tablet_replica.cc:439] T f0d6fccc8fbb4cb1819c89f2296f8838 P ac14eb089ed14bdbbd682f20cb7e9ada: TxnStatusTablet state changed. Reason: New leader ac14eb089ed14bdbbd682f20cb7e9ada. Latest consensus state: current_term: 1 leader_uuid: "ac14eb089ed14bdbbd682f20cb7e9ada" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac14eb089ed14bdbbd682f20cb7e9ada" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42739 } } peers { permanent_uuid: "1f623bcbb73947e8ba2becbff3aa228f" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 41193 } } peers { permanent_uuid: "09ea3124898843e0a7a68d99d83c9f83" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 35133 } } }
I20260808 04:04:03.952481 8596 tablet_replica.cc:442] T f0d6fccc8fbb4cb1819c89f2296f8838 P ac14eb089ed14bdbbd682f20cb7e9ada: This TxnStatusTablet replica's current role is: LEADER
I20260808 04:04:03.953142 8598 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260808 04:04:03.953421 8598 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260808 04:04:03.953389 7874 catalog_manager.cc:5495] T f0d6fccc8fbb4cb1819c89f2296f8838 P ac14eb089ed14bdbbd682f20cb7e9ada reported cstate change: term changed from 0 to 1, leader changed from <none> to ac14eb089ed14bdbbd682f20cb7e9ada (127.5.83.1). New cstate: current_term: 1 leader_uuid: "ac14eb089ed14bdbbd682f20cb7e9ada" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac14eb089ed14bdbbd682f20cb7e9ada" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42739 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f623bcbb73947e8ba2becbff3aa228f" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 41193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09ea3124898843e0a7a68d99d83c9f83" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 35133 } health_report { overall_health: UNKNOWN } } }
I20260808 04:04:03.974099 8558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:03.968309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56428 (local address 127.5.83.1:42739)
0808 04:04:03.968486 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:03.968490 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:03.968552 (+ 62us) server_negotiation.cc:407] Connection header received
0808 04:04:03.968804 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:03.968807 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:03.968882 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:03.968983 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:04:03.969368 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:03.970126 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:03.971982 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:03.972413 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:03.972502 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:03.972997 (+ 495us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:04:03.973502 (+ 505us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:04:03.973647 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:03.973765 (+ 118us) server_negotiation.cc:299] Negotiation successful
0808 04:04:03.973845 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260808 04:04:03.974189 8587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:03.968228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:42739 (local address 127.0.0.1:56428)
0808 04:04:03.968466 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:03.968486 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:03.968635 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:03.969029 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:03.969034 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:03.969046 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 04:04:03.969247 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:03.969258 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:03.970246 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:03.970252 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:03.971856 (+ 1604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:03.971872 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:03.972556 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:03.972562 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:03.972747 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:03.972824 (+ 77us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 04:04:03.973624 (+ 800us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 04:04:03.973646 (+ 22us) client_negotiation.cc:769] Sending connection context
0808 04:04:03.973737 (+ 91us) client_negotiation.cc:240] Negotiation successful
0808 04:04:03.973888 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260808 04:04:03.980808 8283 raft_consensus.cc:1261] T f0d6fccc8fbb4cb1819c89f2296f8838 P 1f623bcbb73947e8ba2becbff3aa228f [term 1 FOLLOWER]: Refusing update from remote peer ac14eb089ed14bdbbd682f20cb7e9ada: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:04:03.980959 8461 raft_consensus.cc:1261] T f0d6fccc8fbb4cb1819c89f2296f8838 P 09ea3124898843e0a7a68d99d83c9f83 [term 1 FOLLOWER]: Refusing update from remote peer ac14eb089ed14bdbbd682f20cb7e9ada: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:04:03.981662 8596 consensus_queue.cc:1035] T f0d6fccc8fbb4cb1819c89f2296f8838 P ac14eb089ed14bdbbd682f20cb7e9ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f623bcbb73947e8ba2becbff3aa228f" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 41193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:04:03.982033 8572 consensus_queue.cc:1035] T f0d6fccc8fbb4cb1819c89f2296f8838 P ac14eb089ed14bdbbd682f20cb7e9ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ea3124898843e0a7a68d99d83c9f83" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 35133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:04:03.992681 8570 tablet_replica.cc:439] T f0d6fccc8fbb4cb1819c89f2296f8838 P 1f623bcbb73947e8ba2becbff3aa228f: TxnStatusTablet state changed. Reason: New leader ac14eb089ed14bdbbd682f20cb7e9ada. Latest consensus state: current_term: 1 leader_uuid: "ac14eb089ed14bdbbd682f20cb7e9ada" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac14eb089ed14bdbbd682f20cb7e9ada" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42739 } } peers { permanent_uuid: "1f623bcbb73947e8ba2becbff3aa228f" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 41193 } } peers { permanent_uuid: "09ea3124898843e0a7a68d99d83c9f83" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 35133 } } }
I20260808 04:04:03.994087 8573 tablet_replica.cc:439] T f0d6fccc8fbb4cb1819c89f2296f8838 P 09ea3124898843e0a7a68d99d83c9f83: TxnStatusTablet state changed. Reason: New leader ac14eb089ed14bdbbd682f20cb7e9ada. Latest consensus state: current_term: 1 leader_uuid: "ac14eb089ed14bdbbd682f20cb7e9ada" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac14eb089ed14bdbbd682f20cb7e9ada" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42739 } } peers { permanent_uuid: "1f623bcbb73947e8ba2becbff3aa228f" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 41193 } } peers { permanent_uuid: "09ea3124898843e0a7a68d99d83c9f83" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 35133 } } }
I20260808 04:04:03.994338 8573 tablet_replica.cc:442] T f0d6fccc8fbb4cb1819c89f2296f8838 P 09ea3124898843e0a7a68d99d83c9f83: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 04:04:03.993005 8570 tablet_replica.cc:442] T f0d6fccc8fbb4cb1819c89f2296f8838 P 1f623bcbb73947e8ba2becbff3aa228f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 04:04:03.997931 8596 tablet_replica.cc:439] T f0d6fccc8fbb4cb1819c89f2296f8838 P ac14eb089ed14bdbbd682f20cb7e9ada: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ac14eb089ed14bdbbd682f20cb7e9ada" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac14eb089ed14bdbbd682f20cb7e9ada" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42739 } } peers { permanent_uuid: "1f623bcbb73947e8ba2becbff3aa228f" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 41193 } } peers { permanent_uuid: "09ea3124898843e0a7a68d99d83c9f83" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 35133 } } }
I20260808 04:04:03.998183 8596 tablet_replica.cc:442] T f0d6fccc8fbb4cb1819c89f2296f8838 P ac14eb089ed14bdbbd682f20cb7e9ada: This TxnStatusTablet replica's current role is: LEADER
I20260808 04:04:03.998405 8596 tablet_replica.cc:439] T f0d6fccc8fbb4cb1819c89f2296f8838 P ac14eb089ed14bdbbd682f20cb7e9ada: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ac14eb089ed14bdbbd682f20cb7e9ada" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac14eb089ed14bdbbd682f20cb7e9ada" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42739 } } peers { permanent_uuid: "1f623bcbb73947e8ba2becbff3aa228f" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 41193 } } peers { permanent_uuid: "09ea3124898843e0a7a68d99d83c9f83" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 35133 } } }
I20260808 04:04:03.998673 8596 tablet_replica.cc:442] T f0d6fccc8fbb4cb1819c89f2296f8838 P ac14eb089ed14bdbbd682f20cb7e9ada: This TxnStatusTablet replica's current role is: LEADER
I20260808 04:04:04.013206 8573 tablet_replica.cc:439] T f0d6fccc8fbb4cb1819c89f2296f8838 P 09ea3124898843e0a7a68d99d83c9f83: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ac14eb089ed14bdbbd682f20cb7e9ada" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac14eb089ed14bdbbd682f20cb7e9ada" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42739 } } peers { permanent_uuid: "1f623bcbb73947e8ba2becbff3aa228f" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 41193 } } peers { permanent_uuid: "09ea3124898843e0a7a68d99d83c9f83" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 35133 } } }
I20260808 04:04:04.013490 8573 tablet_replica.cc:442] T f0d6fccc8fbb4cb1819c89f2296f8838 P 09ea3124898843e0a7a68d99d83c9f83: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 04:04:04.018110 8570 tablet_replica.cc:439] T f0d6fccc8fbb4cb1819c89f2296f8838 P 1f623bcbb73947e8ba2becbff3aa228f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ac14eb089ed14bdbbd682f20cb7e9ada" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac14eb089ed14bdbbd682f20cb7e9ada" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42739 } } peers { permanent_uuid: "1f623bcbb73947e8ba2becbff3aa228f" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 41193 } } peers { permanent_uuid: "09ea3124898843e0a7a68d99d83c9f83" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 35133 } } }
I20260808 04:04:04.018414 8570 tablet_replica.cc:442] T f0d6fccc8fbb4cb1819c89f2296f8838 P 1f623bcbb73947e8ba2becbff3aa228f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 04:04:04.026152 8592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:04.013880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57564 (local address 127.5.83.2:41193)
0808 04:04:04.014103 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:04.014111 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:04.020365 (+ 6254us) server_negotiation.cc:407] Connection header received
0808 04:04:04.020531 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:04.020535 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:04.020606 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:04.020717 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:04:04.021256 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:04.021991 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:04.023994 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:04.024481 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:04.024575 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:04.025055 (+ 480us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:04:04.025577 (+ 522us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:04:04.025679 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:04.025828 (+ 149us) server_negotiation.cc:299] Negotiation successful
0808 04:04:04.025901 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 04:04:04.041479 8624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:04.035765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:42739 (local address 127.5.83.2:40427)
0808 04:04:04.036396 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:04.036413 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:04.036519 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:04.036828 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:04.036831 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:04.036840 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 04:04:04.037102 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:04.037109 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:04.038523 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:04.038526 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:04.039690 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:04.039726 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:04.040415 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:04.040421 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:04.040547 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:04.040600 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 04:04:04.040986 (+ 386us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 04:04:04.040997 (+ 11us) client_negotiation.cc:769] Sending connection context
0808 04:04:04.041068 (+ 71us) client_negotiation.cc:240] Negotiation successful
0808 04:04:04.041190 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":356,"threads_started":1}
I20260808 04:04:04.041483 8558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:04.035816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.2:40427 (local address 127.5.83.1:42739)
0808 04:04:04.035985 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:04.035991 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:04.036486 (+ 495us) server_negotiation.cc:407] Connection header received
0808 04:04:04.036632 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:04.036637 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:04.036725 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:04.036817 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:04:04.037233 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:04.038405 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:04.039846 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:04.040273 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:04.040381 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:04.040773 (+ 392us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:04:04.040874 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:04:04.040991 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:04.041099 (+ 108us) server_negotiation.cc:299] Negotiation successful
0808 04:04:04.041176 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 04:04:04.051087 8461 raft_consensus.cc:1261] T a637aee39ae44ca9a7d320f2c2739fdf P 09ea3124898843e0a7a68d99d83c9f83 [term 1 FOLLOWER]: Refusing update from remote peer 1f623bcbb73947e8ba2becbff3aa228f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:04:04.051309 8091 raft_consensus.cc:1261] T a637aee39ae44ca9a7d320f2c2739fdf P ac14eb089ed14bdbbd682f20cb7e9ada [term 1 FOLLOWER]: Refusing update from remote peer 1f623bcbb73947e8ba2becbff3aa228f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:04:04.051748 8570 consensus_queue.cc:1035] T a637aee39ae44ca9a7d320f2c2739fdf P 1f623bcbb73947e8ba2becbff3aa228f [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ea3124898843e0a7a68d99d83c9f83" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 35133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:04:04.052287 8570 consensus_queue.cc:1035] T a637aee39ae44ca9a7d320f2c2739fdf P 1f623bcbb73947e8ba2becbff3aa228f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac14eb089ed14bdbbd682f20cb7e9ada" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:04:04.073544 8612 mvcc.cc:205] Tried to move back new op lower bound from 7316118913226469376 to 7316118911180865536. Current Snapshot: MvccSnapshot[applied={T|T < 7316118913226469376}]
I20260808 04:04:04.107743 8592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:04.101919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.1:48301 (local address 127.5.83.2:41193)
0808 04:04:04.102078 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:04.102083 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:04.102221 (+ 138us) server_negotiation.cc:407] Connection header received
0808 04:04:04.102459 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:04.102463 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:04.102532 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:04.102602 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:04:04.103205 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:04.104110 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:04.106187 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:04.106568 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:04.106664 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:04.107154 (+ 490us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:04:04.107249 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:04:04.107309 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:04.107452 (+ 143us) server_negotiation.cc:299] Negotiation successful
0808 04:04:04.107511 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 04:04:04.108031 8632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:04.101362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.2:41193 (local address 127.5.83.1:48301)
0808 04:04:04.102117 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:04.102136 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:04.102338 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:04.102637 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:04.102642 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:04.102655 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 04:04:04.103060 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:04.103073 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:04.104229 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:04.104235 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:04.106058 (+ 1823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:04.106074 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:04.106744 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:04.106751 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:04.106939 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:04.107009 (+ 70us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 04:04:04.107337 (+ 328us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 04:04:04.107348 (+ 11us) client_negotiation.cc:769] Sending connection context
0808 04:04:04.107441 (+ 93us) client_negotiation.cc:240] Negotiation successful
0808 04:04:04.107580 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":731,"threads_started":1}
I20260808 04:04:04.144253 7874 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48898:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:04:04.144562 7874 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48898:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:04:04.148478 7874 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4f45675d9036448fbf2bbd0b2d808bc7: Sending DeleteTablet for 3 replicas of tablet a637aee39ae44ca9a7d320f2c2739fdf
I20260808 04:04:04.149945 8057 tablet_service.cc:1514] Processing DeleteTablet for tablet a637aee39ae44ca9a7d320f2c2739fdf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:04:04 UTC) from {username='kudu', principal='kudu/127.5.83.62@KRBTEST.COM'} at 127.0.0.1:56412
I20260808 04:04:04.149966 8436 tablet_service.cc:1514] Processing DeleteTablet for tablet a637aee39ae44ca9a7d320f2c2739fdf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:04:04 UTC) from {username='kudu', principal='kudu/127.5.83.62@KRBTEST.COM'} at 127.0.0.1:41000
I20260808 04:04:04.150774 8249 tablet_service.cc:1514] Processing DeleteTablet for tablet a637aee39ae44ca9a7d320f2c2739fdf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:04:04 UTC) from {username='kudu', principal='kudu/127.5.83.62@KRBTEST.COM'} at 127.0.0.1:57554
I20260808 04:04:04.157603 7874 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48898:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:04:04.158243 7874 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:04:04.163126 8637 tablet_replica.cc:330] T a637aee39ae44ca9a7d320f2c2739fdf P ac14eb089ed14bdbbd682f20cb7e9ada: stopping tablet replica
I20260808 04:04:04.163686 8637 raft_consensus.cc:2229] T a637aee39ae44ca9a7d320f2c2739fdf P ac14eb089ed14bdbbd682f20cb7e9ada [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:04:04.164153 8637 raft_consensus.cc:2258] T a637aee39ae44ca9a7d320f2c2739fdf P ac14eb089ed14bdbbd682f20cb7e9ada [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:04:04.165970 8637 ts_tablet_manager.cc:1905] T a637aee39ae44ca9a7d320f2c2739fdf P ac14eb089ed14bdbbd682f20cb7e9ada: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:04:04.173718 8637 ts_tablet_manager.cc:1918] T a637aee39ae44ca9a7d320f2c2739fdf P ac14eb089ed14bdbbd682f20cb7e9ada: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 04:04:04.173954 8637 log.cc:1196] T a637aee39ae44ca9a7d320f2c2739fdf P ac14eb089ed14bdbbd682f20cb7e9ada: Deleting WAL directory at /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/minicluster-data/ts-0/wal/wals/a637aee39ae44ca9a7d320f2c2739fdf
I20260808 04:04:04.174615 8637 ts_tablet_manager.cc:1939] T a637aee39ae44ca9a7d320f2c2739fdf P ac14eb089ed14bdbbd682f20cb7e9ada: Deleting consensus metadata
I20260808 04:04:04.176705 8057 tablet_service.cc:1467] Processing CreateTablet for tablet be91e7a98d0b476598a43902564cfa81 (DEFAULT_TABLE table=test-table [id=17919115a4394378876e918d6607fb87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:04:04.177018 7859 catalog_manager.cc:4841] TS ac14eb089ed14bdbbd682f20cb7e9ada (127.5.83.1:42739): tablet a637aee39ae44ca9a7d320f2c2739fdf (table test-table [id=ccb1901d4a944307a75aae933e2d1aaa]) successfully deleted
I20260808 04:04:04.177150 8057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be91e7a98d0b476598a43902564cfa81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:04:04.177062 8246 tablet_service.cc:1467] Processing CreateTablet for tablet be91e7a98d0b476598a43902564cfa81 (DEFAULT_TABLE table=test-table [id=17919115a4394378876e918d6607fb87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:04:04.177567 8246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be91e7a98d0b476598a43902564cfa81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:04:04.184145 8435 tablet_service.cc:1467] Processing CreateTablet for tablet be91e7a98d0b476598a43902564cfa81 (DEFAULT_TABLE table=test-table [id=17919115a4394378876e918d6607fb87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:04:04.184681 8435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be91e7a98d0b476598a43902564cfa81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:04:04.193017 8640 tablet_bootstrap.cc:492] T be91e7a98d0b476598a43902564cfa81 P ac14eb089ed14bdbbd682f20cb7e9ada: Bootstrap starting.
I20260808 04:04:04.195322 8640 tablet_bootstrap.cc:654] T be91e7a98d0b476598a43902564cfa81 P ac14eb089ed14bdbbd682f20cb7e9ada: Neither blocks nor log segments found. Creating new log.
I20260808 04:04:04.206251 8641 tablet_bootstrap.cc:492] T be91e7a98d0b476598a43902564cfa81 P 09ea3124898843e0a7a68d99d83c9f83: Bootstrap starting.
I20260808 04:04:04.207690 8638 tablet_replica.cc:330] T a637aee39ae44ca9a7d320f2c2739fdf P 1f623bcbb73947e8ba2becbff3aa228f: stopping tablet replica
I20260808 04:04:04.208101 8641 tablet_bootstrap.cc:654] T be91e7a98d0b476598a43902564cfa81 P 09ea3124898843e0a7a68d99d83c9f83: Neither blocks nor log segments found. Creating new log.
I20260808 04:04:04.208293 8638 raft_consensus.cc:2229] T a637aee39ae44ca9a7d320f2c2739fdf P 1f623bcbb73947e8ba2becbff3aa228f [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:04:04.208877 8638 raft_consensus.cc:2258] T a637aee39ae44ca9a7d320f2c2739fdf P 1f623bcbb73947e8ba2becbff3aa228f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:04:04.212085 8636 tablet_replica.cc:330] T a637aee39ae44ca9a7d320f2c2739fdf P 09ea3124898843e0a7a68d99d83c9f83: stopping tablet replica
I20260808 04:04:04.212440 8636 raft_consensus.cc:2229] T a637aee39ae44ca9a7d320f2c2739fdf P 09ea3124898843e0a7a68d99d83c9f83 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:04:04.212775 8636 raft_consensus.cc:2258] T a637aee39ae44ca9a7d320f2c2739fdf P 09ea3124898843e0a7a68d99d83c9f83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:04:04.216203 8639 tablet_bootstrap.cc:492] T be91e7a98d0b476598a43902564cfa81 P 1f623bcbb73947e8ba2becbff3aa228f: Bootstrap starting.
I20260808 04:04:04.217469 8638 ts_tablet_manager.cc:1905] T a637aee39ae44ca9a7d320f2c2739fdf P 1f623bcbb73947e8ba2becbff3aa228f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:04:04.218135 8639 tablet_bootstrap.cc:654] T be91e7a98d0b476598a43902564cfa81 P 1f623bcbb73947e8ba2becbff3aa228f: Neither blocks nor log segments found. Creating new log.
I20260808 04:04:04.220906 8641 tablet_bootstrap.cc:492] T be91e7a98d0b476598a43902564cfa81 P 09ea3124898843e0a7a68d99d83c9f83: No bootstrap required, opened a new log
I20260808 04:04:04.221066 8641 ts_tablet_manager.cc:1397] T be91e7a98d0b476598a43902564cfa81 P 09ea3124898843e0a7a68d99d83c9f83: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.000s
I20260808 04:04:04.222016 8636 ts_tablet_manager.cc:1905] T a637aee39ae44ca9a7d320f2c2739fdf P 09ea3124898843e0a7a68d99d83c9f83: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:04:04.221902 8641 raft_consensus.cc:348] T be91e7a98d0b476598a43902564cfa81 P 09ea3124898843e0a7a68d99d83c9f83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac14eb089ed14bdbbd682f20cb7e9ada" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42739 } } peers { permanent_uuid: "1f623bcbb73947e8ba2becbff3aa228f" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 41193 } } peers { permanent_uuid: "09ea3124898843e0a7a68d99d83c9f83" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 35133 } }
I20260808 04:04:04.222143 8641 raft_consensus.cc:374] T be91e7a98d0b476598a43902564cfa81 P 09ea3124898843e0a7a68d99d83c9f83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:04:04.222234 8641 raft_consensus.cc:729] T be91e7a98d0b476598a43902564cfa81 P 09ea3124898843e0a7a68d99d83c9f83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ea3124898843e0a7a68d99d83c9f83, State: Initialized, Role: FOLLOWER
I20260808 04:04:04.222554 8641 consensus_queue.cc:260] T be91e7a98d0b476598a43902564cfa81 P 09ea3124898843e0a7a68d99d83c9f83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac14eb089ed14bdbbd682f20cb7e9ada" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42739 } } peers { permanent_uuid: "1f623bcbb73947e8ba2becbff3aa228f" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 41193 } } peers { permanent_uuid: "09ea3124898843e0a7a68d99d83c9f83" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 35133 } }
I20260808 04:04:04.230170 8641 ts_tablet_manager.cc:1428] T be91e7a98d0b476598a43902564cfa81 P 09ea3124898843e0a7a68d99d83c9f83: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20260808 04:04:04.234231 8640 tablet_bootstrap.cc:492] T be91e7a98d0b476598a43902564cfa81 P ac14eb089ed14bdbbd682f20cb7e9ada: No bootstrap required, opened a new log
I20260808 04:04:04.237636 8636 ts_tablet_manager.cc:1918] T a637aee39ae44ca9a7d320f2c2739fdf P 09ea3124898843e0a7a68d99d83c9f83: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 04:04:04.237783 8636 log.cc:1196] T a637aee39ae44ca9a7d320f2c2739fdf P 09ea3124898843e0a7a68d99d83c9f83: Deleting WAL directory at /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/minicluster-data/ts-2/wal/wals/a637aee39ae44ca9a7d320f2c2739fdf
I20260808 04:04:04.238260 8636 ts_tablet_manager.cc:1939] T a637aee39ae44ca9a7d320f2c2739fdf P 09ea3124898843e0a7a68d99d83c9f83: Deleting consensus metadata
I20260808 04:04:04.238435 8638 ts_tablet_manager.cc:1918] T a637aee39ae44ca9a7d320f2c2739fdf P 1f623bcbb73947e8ba2becbff3aa228f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 04:04:04.238587 8638 log.cc:1196] T a637aee39ae44ca9a7d320f2c2739fdf P 1f623bcbb73947e8ba2becbff3aa228f: Deleting WAL directory at /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/minicluster-data/ts-1/wal/wals/a637aee39ae44ca9a7d320f2c2739fdf
I20260808 04:04:04.238996 8638 ts_tablet_manager.cc:1939] T a637aee39ae44ca9a7d320f2c2739fdf P 1f623bcbb73947e8ba2becbff3aa228f: Deleting consensus metadata
I20260808 04:04:04.239985 7859 catalog_manager.cc:4841] TS 09ea3124898843e0a7a68d99d83c9f83 (127.5.83.3:35133): tablet a637aee39ae44ca9a7d320f2c2739fdf (table test-table [id=ccb1901d4a944307a75aae933e2d1aaa]) successfully deleted
I20260808 04:04:04.240486 7859 catalog_manager.cc:4841] TS 1f623bcbb73947e8ba2becbff3aa228f (127.5.83.2:41193): tablet a637aee39ae44ca9a7d320f2c2739fdf (table test-table [id=ccb1901d4a944307a75aae933e2d1aaa]) successfully deleted
I20260808 04:04:04.242240 8639 tablet_bootstrap.cc:492] T be91e7a98d0b476598a43902564cfa81 P 1f623bcbb73947e8ba2becbff3aa228f: No bootstrap required, opened a new log
I20260808 04:04:04.242453 8639 ts_tablet_manager.cc:1397] T be91e7a98d0b476598a43902564cfa81 P 1f623bcbb73947e8ba2becbff3aa228f: Time spent bootstrapping tablet: real 0.026s user 0.000s sys 0.009s
I20260808 04:04:04.243292 8639 raft_consensus.cc:348] T be91e7a98d0b476598a43902564cfa81 P 1f623bcbb73947e8ba2becbff3aa228f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac14eb089ed14bdbbd682f20cb7e9ada" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42739 } } peers { permanent_uuid: "1f623bcbb73947e8ba2becbff3aa228f" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 41193 } } peers { permanent_uuid: "09ea3124898843e0a7a68d99d83c9f83" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 35133 } }
I20260808 04:04:04.243587 8639 raft_consensus.cc:374] T be91e7a98d0b476598a43902564cfa81 P 1f623bcbb73947e8ba2becbff3aa228f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:04:04.243760 8639 raft_consensus.cc:729] T be91e7a98d0b476598a43902564cfa81 P 1f623bcbb73947e8ba2becbff3aa228f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f623bcbb73947e8ba2becbff3aa228f, State: Initialized, Role: FOLLOWER
I20260808 04:04:04.244061 8639 consensus_queue.cc:260] T be91e7a98d0b476598a43902564cfa81 P 1f623bcbb73947e8ba2becbff3aa228f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac14eb089ed14bdbbd682f20cb7e9ada" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42739 } } peers { permanent_uuid: "1f623bcbb73947e8ba2becbff3aa228f" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 41193 } } peers { permanent_uuid: "09ea3124898843e0a7a68d99d83c9f83" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 35133 } }
I20260808 04:04:04.244649 8640 ts_tablet_manager.cc:1397] T be91e7a98d0b476598a43902564cfa81 P ac14eb089ed14bdbbd682f20cb7e9ada: Time spent bootstrapping tablet: real 0.044s user 0.019s sys 0.019s
I20260808 04:04:04.245355 8639 ts_tablet_manager.cc:1428] T be91e7a98d0b476598a43902564cfa81 P 1f623bcbb73947e8ba2becbff3aa228f: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260808 04:04:04.245573 8640 raft_consensus.cc:348] T be91e7a98d0b476598a43902564cfa81 P ac14eb089ed14bdbbd682f20cb7e9ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac14eb089ed14bdbbd682f20cb7e9ada" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42739 } } peers { permanent_uuid: "1f623bcbb73947e8ba2becbff3aa228f" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 41193 } } peers { permanent_uuid: "09ea3124898843e0a7a68d99d83c9f83" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 35133 } }
I20260808 04:04:04.245850 8640 raft_consensus.cc:374] T be91e7a98d0b476598a43902564cfa81 P ac14eb089ed14bdbbd682f20cb7e9ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:04:04.245957 8640 raft_consensus.cc:729] T be91e7a98d0b476598a43902564cfa81 P ac14eb089ed14bdbbd682f20cb7e9ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac14eb089ed14bdbbd682f20cb7e9ada, State: Initialized, Role: FOLLOWER
I20260808 04:04:04.246204 8640 consensus_queue.cc:260] T be91e7a98d0b476598a43902564cfa81 P ac14eb089ed14bdbbd682f20cb7e9ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac14eb089ed14bdbbd682f20cb7e9ada" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42739 } } peers { permanent_uuid: "1f623bcbb73947e8ba2becbff3aa228f" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 41193 } } peers { permanent_uuid: "09ea3124898843e0a7a68d99d83c9f83" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 35133 } }
I20260808 04:04:04.246745 8640 ts_tablet_manager.cc:1428] T be91e7a98d0b476598a43902564cfa81 P ac14eb089ed14bdbbd682f20cb7e9ada: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260808 04:04:04.481339 8573 raft_consensus.cc:482] T be91e7a98d0b476598a43902564cfa81 P 09ea3124898843e0a7a68d99d83c9f83 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:04:04.481566 8573 raft_consensus.cc:504] T be91e7a98d0b476598a43902564cfa81 P 09ea3124898843e0a7a68d99d83c9f83 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac14eb089ed14bdbbd682f20cb7e9ada" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42739 } } peers { permanent_uuid: "1f623bcbb73947e8ba2becbff3aa228f" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 41193 } } peers { permanent_uuid: "09ea3124898843e0a7a68d99d83c9f83" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 35133 } }
I20260808 04:04:04.483026 8573 leader_election.cc:288] T be91e7a98d0b476598a43902564cfa81 P 09ea3124898843e0a7a68d99d83c9f83 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac14eb089ed14bdbbd682f20cb7e9ada (127.5.83.1:42739), 1f623bcbb73947e8ba2becbff3aa228f (127.5.83.2:41193)
I20260808 04:04:04.490409 8558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:04.483424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.3:52019 (local address 127.5.83.1:42739)
0808 04:04:04.483591 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:04.483598 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:04.483975 (+ 377us) server_negotiation.cc:407] Connection header received
0808 04:04:04.484209 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:04.484215 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:04.484294 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:04.484357 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:04:04.485121 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:04.486071 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:04.488250 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:04.489482 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:04.489642 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:04.489740 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:04.490005 (+ 265us) server_negotiation.cc:299] Negotiation successful
0808 04:04:04.490137 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 04:04:04.490761 8646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:04.483364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:42739 (local address 127.5.83.3:52019)
0808 04:04:04.483875 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:04.483891 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:04.484028 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:04.484587 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:04.484592 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:04.484621 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:04:04.484903 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:04.484917 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:04.486187 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:04.486192 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:04.488072 (+ 1880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:04.488087 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:04.489663 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:04.489669 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:04.489829 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:04.489864 (+ 35us) client_negotiation.cc:769] Sending connection context
0808 04:04:04.489963 (+ 99us) client_negotiation.cc:240] Negotiation successful
0808 04:04:04.490089 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"mutex_wait_us":21,"thread_start_us":267,"threads_started":1}
I20260808 04:04:04.491598 8648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:04.484108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.2:41193 (local address 127.5.83.3:45381)
0808 04:04:04.484938 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:04.484957 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:04.485064 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:04.485416 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:04.485421 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:04.485447 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:04:04.485760 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:04.485769 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:04.487194 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:04.487200 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:04.488934 (+ 1734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:04.488946 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:04.490864 (+ 1918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:04.490872 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:04.491030 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:04.491054 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 04:04:04.491133 (+ 79us) client_negotiation.cc:240] Negotiation successful
0808 04:04:04.491243 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":706,"mutex_wait_us":1,"thread_start_us":488,"threads_started":1}
I20260808 04:04:04.491595 8592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:04.484231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.3:45381 (local address 127.5.83.2:41193)
0808 04:04:04.484447 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:04.484453 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:04.485045 (+ 592us) server_negotiation.cc:407] Connection header received
0808 04:04:04.485196 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:04.485201 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:04.485286 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:04.485392 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:04:04.485930 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:04.487060 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:04.489134 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:04.490377 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:04.490501 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:04.490606 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:04.491188 (+ 582us) server_negotiation.cc:299] Negotiation successful
0808 04:04:04.491316 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 04:04:04.491564 8091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be91e7a98d0b476598a43902564cfa81" candidate_uuid: "09ea3124898843e0a7a68d99d83c9f83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac14eb089ed14bdbbd682f20cb7e9ada" is_pre_election: true
I20260808 04:04:04.492013 8091 raft_consensus.cc:2454] T be91e7a98d0b476598a43902564cfa81 P ac14eb089ed14bdbbd682f20cb7e9ada [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09ea3124898843e0a7a68d99d83c9f83 in term 0.
I20260808 04:04:04.492555 8375 leader_election.cc:302] T be91e7a98d0b476598a43902564cfa81 P 09ea3124898843e0a7a68d99d83c9f83 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ea3124898843e0a7a68d99d83c9f83, ac14eb089ed14bdbbd682f20cb7e9ada; no voters:
I20260808 04:04:04.493017 8573 raft_consensus.cc:2781] T be91e7a98d0b476598a43902564cfa81 P 09ea3124898843e0a7a68d99d83c9f83 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:04:04.493134 8573 raft_consensus.cc:482] T be91e7a98d0b476598a43902564cfa81 P 09ea3124898843e0a7a68d99d83c9f83 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:04:04.493228 8573 raft_consensus.cc:3037] T be91e7a98d0b476598a43902564cfa81 P 09ea3124898843e0a7a68d99d83c9f83 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:04:04.493326 8283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be91e7a98d0b476598a43902564cfa81" candidate_uuid: "09ea3124898843e0a7a68d99d83c9f83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f623bcbb73947e8ba2becbff3aa228f" is_pre_election: true
I20260808 04:04:04.493574 8283 raft_consensus.cc:2454] T be91e7a98d0b476598a43902564cfa81 P 1f623bcbb73947e8ba2becbff3aa228f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09ea3124898843e0a7a68d99d83c9f83 in term 0.
I20260808 04:04:04.495539 8573 raft_consensus.cc:504] T be91e7a98d0b476598a43902564cfa81 P 09ea3124898843e0a7a68d99d83c9f83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac14eb089ed14bdbbd682f20cb7e9ada" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42739 } } peers { permanent_uuid: "1f623bcbb73947e8ba2becbff3aa228f" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 41193 } } peers { permanent_uuid: "09ea3124898843e0a7a68d99d83c9f83" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 35133 } }
I20260808 04:04:04.496065 8573 leader_election.cc:288] T be91e7a98d0b476598a43902564cfa81 P 09ea3124898843e0a7a68d99d83c9f83 [CANDIDATE]: Term 1 election: Requested vote from peers ac14eb089ed14bdbbd682f20cb7e9ada (127.5.83.1:42739), 1f623bcbb73947e8ba2becbff3aa228f (127.5.83.2:41193)
I20260808 04:04:04.496680 8091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be91e7a98d0b476598a43902564cfa81" candidate_uuid: "09ea3124898843e0a7a68d99d83c9f83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac14eb089ed14bdbbd682f20cb7e9ada"
I20260808 04:04:04.496701 8283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be91e7a98d0b476598a43902564cfa81" candidate_uuid: "09ea3124898843e0a7a68d99d83c9f83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f623bcbb73947e8ba2becbff3aa228f"
I20260808 04:04:04.496855 8091 raft_consensus.cc:3037] T be91e7a98d0b476598a43902564cfa81 P ac14eb089ed14bdbbd682f20cb7e9ada [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:04:04.496896 8283 raft_consensus.cc:3037] T be91e7a98d0b476598a43902564cfa81 P 1f623bcbb73947e8ba2becbff3aa228f [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:04:04.498950 8091 raft_consensus.cc:2454] T be91e7a98d0b476598a43902564cfa81 P ac14eb089ed14bdbbd682f20cb7e9ada [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ea3124898843e0a7a68d99d83c9f83 in term 1.
I20260808 04:04:04.499369 8375 leader_election.cc:302] T be91e7a98d0b476598a43902564cfa81 P 09ea3124898843e0a7a68d99d83c9f83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ea3124898843e0a7a68d99d83c9f83, ac14eb089ed14bdbbd682f20cb7e9ada; no voters:
I20260808 04:04:04.499624 8573 raft_consensus.cc:2781] T be91e7a98d0b476598a43902564cfa81 P 09ea3124898843e0a7a68d99d83c9f83 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:04:04.500124 8283 raft_consensus.cc:2454] T be91e7a98d0b476598a43902564cfa81 P 1f623bcbb73947e8ba2becbff3aa228f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ea3124898843e0a7a68d99d83c9f83 in term 1.
I20260808 04:04:04.500125 8573 raft_consensus.cc:686] T be91e7a98d0b476598a43902564cfa81 P 09ea3124898843e0a7a68d99d83c9f83 [term 1 LEADER]: Becoming Leader. State: Replica: 09ea3124898843e0a7a68d99d83c9f83, State: Running, Role: LEADER
I20260808 04:04:04.500751 8573 consensus_queue.cc:237] T be91e7a98d0b476598a43902564cfa81 P 09ea3124898843e0a7a68d99d83c9f83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac14eb089ed14bdbbd682f20cb7e9ada" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42739 } } peers { permanent_uuid: "1f623bcbb73947e8ba2becbff3aa228f" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 41193 } } peers { permanent_uuid: "09ea3124898843e0a7a68d99d83c9f83" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 35133 } }
I20260808 04:04:04.504314 7875 catalog_manager.cc:5495] T be91e7a98d0b476598a43902564cfa81 P 09ea3124898843e0a7a68d99d83c9f83 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09ea3124898843e0a7a68d99d83c9f83 (127.5.83.3). New cstate: current_term: 1 leader_uuid: "09ea3124898843e0a7a68d99d83c9f83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac14eb089ed14bdbbd682f20cb7e9ada" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42739 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f623bcbb73947e8ba2becbff3aa228f" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 41193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09ea3124898843e0a7a68d99d83c9f83" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 35133 } health_report { overall_health: HEALTHY } } }
I20260808 04:04:04.568711 8656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:04.561709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41016 (local address 127.5.83.3:35133)
0808 04:04:04.562918 (+ 1209us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:04.562921 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:04.562939 (+ 18us) server_negotiation.cc:407] Connection header received
0808 04:04:04.563010 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:04.563014 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:04.563093 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:04.563224 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:04:04.563692 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:04.564509 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:04.566462 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:04.566849 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:04.566934 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:04.567441 (+ 507us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:04:04.568064 (+ 623us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:04:04.568166 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:04.568307 (+ 141us) server_negotiation.cc:299] Negotiation successful
0808 04:04:04.568368 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1108,"thread_start_us":992,"threads_started":1}
I20260808 04:04:04.584651 8558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:04.578729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.3:39835 (local address 127.5.83.1:42739)
0808 04:04:04.578868 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:04.578872 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:04.579490 (+ 618us) server_negotiation.cc:407] Connection header received
0808 04:04:04.579609 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:04.579614 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:04.579700 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:04.579818 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:04:04.580352 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:04.581453 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:04.582763 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:04.583389 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:04.583515 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:04.583975 (+ 460us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:04:04.584077 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:04:04.584153 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:04.584286 (+ 133us) server_negotiation.cc:299] Negotiation successful
0808 04:04:04.584365 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 04:04:04.584738 8663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:04.578766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:42739 (local address 127.5.83.3:39835)
0808 04:04:04.579383 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:04.579398 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:04.579512 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:04.579821 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:04.579824 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:04.579834 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 04:04:04.580198 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:04.580211 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:04.581571 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:04.581575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:04.582628 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:04.582638 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:04.583547 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:04.583551 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:04.583665 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:04.583749 (+ 84us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 04:04:04.584173 (+ 424us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 04:04:04.584185 (+ 12us) client_negotiation.cc:769] Sending connection context
0808 04:04:04.584257 (+ 72us) client_negotiation.cc:240] Negotiation successful
0808 04:04:04.584382 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":343,"threads_started":1}
I20260808 04:04:04.592833 8283 raft_consensus.cc:1261] T be91e7a98d0b476598a43902564cfa81 P 1f623bcbb73947e8ba2becbff3aa228f [term 1 FOLLOWER]: Refusing update from remote peer 09ea3124898843e0a7a68d99d83c9f83: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:04:04.593083 8091 raft_consensus.cc:1261] T be91e7a98d0b476598a43902564cfa81 P ac14eb089ed14bdbbd682f20cb7e9ada [term 1 FOLLOWER]: Refusing update from remote peer 09ea3124898843e0a7a68d99d83c9f83: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:04:04.593854 8649 consensus_queue.cc:1035] T be91e7a98d0b476598a43902564cfa81 P 09ea3124898843e0a7a68d99d83c9f83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac14eb089ed14bdbbd682f20cb7e9ada" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:04:04.594696 8649 consensus_queue.cc:1035] T be91e7a98d0b476598a43902564cfa81 P 09ea3124898843e0a7a68d99d83c9f83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f623bcbb73947e8ba2becbff3aa228f" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 41193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:04:04.607982 8610 mvcc.cc:205] Tried to move back new op lower bound from 7316118915445166080 to 7316118915078705152. Current Snapshot: MvccSnapshot[applied={T|T < 7316118915445166080}]
I20260808 04:04:04.611317 8603 mvcc.cc:205] Tried to move back new op lower bound from 7316118915445166080 to 7316118915078705152. Current Snapshot: MvccSnapshot[applied={T|T < 7316118915445166080}]
I20260808 04:04:04.638863 8656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:04.632892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.1:46773 (local address 127.5.83.3:35133)
0808 04:04:04.633088 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:04.633093 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:04.633582 (+ 489us) server_negotiation.cc:407] Connection header received
0808 04:04:04.633865 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:04.633871 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:04.633954 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:04.634079 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:04:04.634491 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:04.635216 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:04.637086 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:04.637719 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:04.637837 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:04.638262 (+ 425us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:04:04.638369 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:04:04.638439 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:04.638570 (+ 131us) server_negotiation.cc:299] Negotiation successful
0808 04:04:04.638630 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260808 04:04:04.639034 8668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:04.632729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.3:35133 (local address 127.5.83.1:46773)
0808 04:04:04.633488 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:04.633507 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:04.633677 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:04.634103 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:04.634108 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:04.634121 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 04:04:04.634334 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:04.634345 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:04.635345 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:04.635351 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:04.636911 (+ 1560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:04.636927 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:04.637875 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:04.637882 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:04.638072 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:04.638124 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 04:04:04.638465 (+ 341us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 04:04:04.638476 (+ 11us) client_negotiation.cc:769] Sending connection context
0808 04:04:04.638554 (+ 78us) client_negotiation.cc:240] Negotiation successful
0808 04:04:04.638690 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":668,"threads_started":1}
W20260808 04:04:04.640213 8610 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260808 04:04:06.261229 7875 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48898:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:04:06.261461 7875 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48898:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:04:06.265396 7875 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4f45675d9036448fbf2bbd0b2d808bc7: Sending DeleteTablet for 3 replicas of tablet be91e7a98d0b476598a43902564cfa81
I20260808 04:04:06.266140 8057 tablet_service.cc:1514] Processing DeleteTablet for tablet be91e7a98d0b476598a43902564cfa81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:04:06 UTC) from {username='kudu', principal='kudu/127.5.83.62@KRBTEST.COM'} at 127.0.0.1:56412
I20260808 04:04:06.266283 8246 tablet_service.cc:1514] Processing DeleteTablet for tablet be91e7a98d0b476598a43902564cfa81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:04:06 UTC) from {username='kudu', principal='kudu/127.5.83.62@KRBTEST.COM'} at 127.0.0.1:57554
I20260808 04:04:06.266736 8436 tablet_service.cc:1514] Processing DeleteTablet for tablet be91e7a98d0b476598a43902564cfa81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:04:06 UTC) from {username='kudu', principal='kudu/127.5.83.62@KRBTEST.COM'} at 127.0.0.1:41000
I20260808 04:04:06.279249 8706 tablet_replica.cc:330] T be91e7a98d0b476598a43902564cfa81 P 1f623bcbb73947e8ba2becbff3aa228f: stopping tablet replica
I20260808 04:04:06.279458 8706 raft_consensus.cc:2229] T be91e7a98d0b476598a43902564cfa81 P 1f623bcbb73947e8ba2becbff3aa228f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:04:06.279718 8706 raft_consensus.cc:2258] T be91e7a98d0b476598a43902564cfa81 P 1f623bcbb73947e8ba2becbff3aa228f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:04:06.280730 8705 tablet_replica.cc:330] T be91e7a98d0b476598a43902564cfa81 P ac14eb089ed14bdbbd682f20cb7e9ada: stopping tablet replica
I20260808 04:04:06.280707 8706 ts_tablet_manager.cc:1905] T be91e7a98d0b476598a43902564cfa81 P 1f623bcbb73947e8ba2becbff3aa228f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:04:06.280917 8705 raft_consensus.cc:2229] T be91e7a98d0b476598a43902564cfa81 P ac14eb089ed14bdbbd682f20cb7e9ada [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:04:06.281147 8705 raft_consensus.cc:2258] T be91e7a98d0b476598a43902564cfa81 P ac14eb089ed14bdbbd682f20cb7e9ada [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:04:06.282112 8705 ts_tablet_manager.cc:1905] T be91e7a98d0b476598a43902564cfa81 P ac14eb089ed14bdbbd682f20cb7e9ada: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:04:06.287132 8708 tablet_replica.cc:330] T be91e7a98d0b476598a43902564cfa81 P 09ea3124898843e0a7a68d99d83c9f83: stopping tablet replica
I20260808 04:04:06.287297 8706 ts_tablet_manager.cc:1918] T be91e7a98d0b476598a43902564cfa81 P 1f623bcbb73947e8ba2becbff3aa228f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 04:04:06.287348 8708 raft_consensus.cc:2229] T be91e7a98d0b476598a43902564cfa81 P 09ea3124898843e0a7a68d99d83c9f83 [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:04:06.287436 8706 log.cc:1196] T be91e7a98d0b476598a43902564cfa81 P 1f623bcbb73947e8ba2becbff3aa228f: Deleting WAL directory at /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/minicluster-data/ts-1/wal/wals/be91e7a98d0b476598a43902564cfa81
I20260808 04:04:06.287940 8706 ts_tablet_manager.cc:1939] T be91e7a98d0b476598a43902564cfa81 P 1f623bcbb73947e8ba2becbff3aa228f: Deleting consensus metadata
I20260808 04:04:06.288008 8708 raft_consensus.cc:2258] T be91e7a98d0b476598a43902564cfa81 P 09ea3124898843e0a7a68d99d83c9f83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:04:06.288976 8708 ts_tablet_manager.cc:1905] T be91e7a98d0b476598a43902564cfa81 P 09ea3124898843e0a7a68d99d83c9f83: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:04:06.289222 7859 catalog_manager.cc:4841] TS 1f623bcbb73947e8ba2becbff3aa228f (127.5.83.2:41193): tablet be91e7a98d0b476598a43902564cfa81 (table test-table [id=17919115a4394378876e918d6607fb87]) successfully deleted
I20260808 04:04:06.289937 8705 ts_tablet_manager.cc:1918] T be91e7a98d0b476598a43902564cfa81 P ac14eb089ed14bdbbd682f20cb7e9ada: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 04:04:06.290138 8705 log.cc:1196] T be91e7a98d0b476598a43902564cfa81 P ac14eb089ed14bdbbd682f20cb7e9ada: Deleting WAL directory at /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/minicluster-data/ts-0/wal/wals/be91e7a98d0b476598a43902564cfa81
I20260808 04:04:06.290644 8705 ts_tablet_manager.cc:1939] T be91e7a98d0b476598a43902564cfa81 P ac14eb089ed14bdbbd682f20cb7e9ada: Deleting consensus metadata
I20260808 04:04:06.291746 7859 catalog_manager.cc:4841] TS ac14eb089ed14bdbbd682f20cb7e9ada (127.5.83.1:42739): tablet be91e7a98d0b476598a43902564cfa81 (table test-table [id=17919115a4394378876e918d6607fb87]) successfully deleted
I20260808 04:04:06.294638 8708 ts_tablet_manager.cc:1918] T be91e7a98d0b476598a43902564cfa81 P 09ea3124898843e0a7a68d99d83c9f83: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 04:04:06.294766 8708 log.cc:1196] T be91e7a98d0b476598a43902564cfa81 P 09ea3124898843e0a7a68d99d83c9f83: Deleting WAL directory at /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/minicluster-data/ts-2/wal/wals/be91e7a98d0b476598a43902564cfa81
I20260808 04:04:06.295176 8708 ts_tablet_manager.cc:1939] T be91e7a98d0b476598a43902564cfa81 P 09ea3124898843e0a7a68d99d83c9f83: Deleting consensus metadata
I20260808 04:04:06.296334 7859 catalog_manager.cc:4841] TS 09ea3124898843e0a7a68d99d83c9f83 (127.5.83.3:35133): tablet be91e7a98d0b476598a43902564cfa81 (table test-table [id=17919115a4394378876e918d6607fb87]) successfully deleted
Aug 08 04:04:06 dist-test-slave-1l3l krb5kdc[7810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161846, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 04:04:06 dist-test-slave-1l3l krb5kdc[7810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161846, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:04:06.325992 8725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:06.313120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48922 (local address 127.5.83.62:45715)
0808 04:04:06.313641 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:06.313645 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:06.313668 (+ 23us) server_negotiation.cc:407] Connection header received
0808 04:04:06.313814 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:06.313818 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:06.313881 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:06.313974 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:06.315186 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:06.316055 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:06.317254 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:06.317652 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:06.317739 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:06.321452 (+ 3713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:06.321457 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:06.321460 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:06.321490 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:06.323950 (+ 2460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:06.324553 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:06.324556 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:06.324559 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:06.324678 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:06.324950 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:06.324953 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:06.324955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:06.325201 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:06.325366 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:06.325586 (+ 220us) server_negotiation.cc:299] Negotiation successful
0808 04:04:06.325737 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":266,"threads_started":1}
I20260808 04:04:06.329173 7875 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48922:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:04:06.329727 7875 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:04:06.335897 8246 tablet_service.cc:1467] Processing CreateTablet for tablet 4755e613b45d4fdaabc4afc8b96034f5 (DEFAULT_TABLE table=test-table [id=3a6a8887f89d4b328d11f53d80ac8167]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:04:06.335897 8057 tablet_service.cc:1467] Processing CreateTablet for tablet 4755e613b45d4fdaabc4afc8b96034f5 (DEFAULT_TABLE table=test-table [id=3a6a8887f89d4b328d11f53d80ac8167]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:04:06.336336 8057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4755e613b45d4fdaabc4afc8b96034f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:04:06.336349 8246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4755e613b45d4fdaabc4afc8b96034f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:04:06.336421 8436 tablet_service.cc:1467] Processing CreateTablet for tablet 4755e613b45d4fdaabc4afc8b96034f5 (DEFAULT_TABLE table=test-table [id=3a6a8887f89d4b328d11f53d80ac8167]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:04:06.336839 8436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4755e613b45d4fdaabc4afc8b96034f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:04:06.341704 8727 tablet_bootstrap.cc:492] T 4755e613b45d4fdaabc4afc8b96034f5 P 09ea3124898843e0a7a68d99d83c9f83: Bootstrap starting.
I20260808 04:04:06.343026 8728 tablet_bootstrap.cc:492] T 4755e613b45d4fdaabc4afc8b96034f5 P 1f623bcbb73947e8ba2becbff3aa228f: Bootstrap starting.
I20260808 04:04:06.343477 8727 tablet_bootstrap.cc:654] T 4755e613b45d4fdaabc4afc8b96034f5 P 09ea3124898843e0a7a68d99d83c9f83: Neither blocks nor log segments found. Creating new log.
I20260808 04:04:06.344888 8728 tablet_bootstrap.cc:654] T 4755e613b45d4fdaabc4afc8b96034f5 P 1f623bcbb73947e8ba2becbff3aa228f: Neither blocks nor log segments found. Creating new log.
I20260808 04:04:06.346225 8727 tablet_bootstrap.cc:492] T 4755e613b45d4fdaabc4afc8b96034f5 P 09ea3124898843e0a7a68d99d83c9f83: No bootstrap required, opened a new log
I20260808 04:04:06.346341 8727 ts_tablet_manager.cc:1397] T 4755e613b45d4fdaabc4afc8b96034f5 P 09ea3124898843e0a7a68d99d83c9f83: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260808 04:04:06.346930 8727 raft_consensus.cc:348] T 4755e613b45d4fdaabc4afc8b96034f5 P 09ea3124898843e0a7a68d99d83c9f83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac14eb089ed14bdbbd682f20cb7e9ada" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42739 } } peers { permanent_uuid: "1f623bcbb73947e8ba2becbff3aa228f" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 41193 } } peers { permanent_uuid: "09ea3124898843e0a7a68d99d83c9f83" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 35133 } }
I20260808 04:04:06.347143 8727 raft_consensus.cc:374] T 4755e613b45d4fdaabc4afc8b96034f5 P 09ea3124898843e0a7a68d99d83c9f83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:04:06.347252 8727 raft_consensus.cc:729] T 4755e613b45d4fdaabc4afc8b96034f5 P 09ea3124898843e0a7a68d99d83c9f83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ea3124898843e0a7a68d99d83c9f83, State: Initialized, Role: FOLLOWER
I20260808 04:04:06.347496 8727 consensus_queue.cc:260] T 4755e613b45d4fdaabc4afc8b96034f5 P 09ea3124898843e0a7a68d99d83c9f83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac14eb089ed14bdbbd682f20cb7e9ada" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42739 } } peers { permanent_uuid: "1f623bcbb73947e8ba2becbff3aa228f" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 41193 } } peers { permanent_uuid: "09ea3124898843e0a7a68d99d83c9f83" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 35133 } }
I20260808 04:04:06.348003 8727 ts_tablet_manager.cc:1428] T 4755e613b45d4fdaabc4afc8b96034f5 P 09ea3124898843e0a7a68d99d83c9f83: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 04:04:06.349145 8729 tablet_bootstrap.cc:492] T 4755e613b45d4fdaabc4afc8b96034f5 P ac14eb089ed14bdbbd682f20cb7e9ada: Bootstrap starting.
I20260808 04:04:06.349342 8728 tablet_bootstrap.cc:492] T 4755e613b45d4fdaabc4afc8b96034f5 P 1f623bcbb73947e8ba2becbff3aa228f: No bootstrap required, opened a new log
I20260808 04:04:06.349475 8728 ts_tablet_manager.cc:1397] T 4755e613b45d4fdaabc4afc8b96034f5 P 1f623bcbb73947e8ba2becbff3aa228f: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.003s
I20260808 04:04:06.350088 8728 raft_consensus.cc:348] T 4755e613b45d4fdaabc4afc8b96034f5 P 1f623bcbb73947e8ba2becbff3aa228f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac14eb089ed14bdbbd682f20cb7e9ada" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42739 } } peers { permanent_uuid: "1f623bcbb73947e8ba2becbff3aa228f" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 41193 } } peers { permanent_uuid: "09ea3124898843e0a7a68d99d83c9f83" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 35133 } }
I20260808 04:04:06.350317 8728 raft_consensus.cc:374] T 4755e613b45d4fdaabc4afc8b96034f5 P 1f623bcbb73947e8ba2becbff3aa228f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:04:06.350417 8728 raft_consensus.cc:729] T 4755e613b45d4fdaabc4afc8b96034f5 P 1f623bcbb73947e8ba2becbff3aa228f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f623bcbb73947e8ba2becbff3aa228f, State: Initialized, Role: FOLLOWER
I20260808 04:04:06.350638 8728 consensus_queue.cc:260] T 4755e613b45d4fdaabc4afc8b96034f5 P 1f623bcbb73947e8ba2becbff3aa228f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac14eb089ed14bdbbd682f20cb7e9ada" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42739 } } peers { permanent_uuid: "1f623bcbb73947e8ba2becbff3aa228f" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 41193 } } peers { permanent_uuid: "09ea3124898843e0a7a68d99d83c9f83" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 35133 } }
I20260808 04:04:06.350878 8729 tablet_bootstrap.cc:654] T 4755e613b45d4fdaabc4afc8b96034f5 P ac14eb089ed14bdbbd682f20cb7e9ada: Neither blocks nor log segments found. Creating new log.
I20260808 04:04:06.352494 8728 ts_tablet_manager.cc:1428] T 4755e613b45d4fdaabc4afc8b96034f5 P 1f623bcbb73947e8ba2becbff3aa228f: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260808 04:04:06.353675 8729 tablet_bootstrap.cc:492] T 4755e613b45d4fdaabc4afc8b96034f5 P ac14eb089ed14bdbbd682f20cb7e9ada: No bootstrap required, opened a new log
I20260808 04:04:06.353792 8729 ts_tablet_manager.cc:1397] T 4755e613b45d4fdaabc4afc8b96034f5 P ac14eb089ed14bdbbd682f20cb7e9ada: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260808 04:04:06.354509 8729 raft_consensus.cc:348] T 4755e613b45d4fdaabc4afc8b96034f5 P ac14eb089ed14bdbbd682f20cb7e9ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac14eb089ed14bdbbd682f20cb7e9ada" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42739 } } peers { permanent_uuid: "1f623bcbb73947e8ba2becbff3aa228f" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 41193 } } peers { permanent_uuid: "09ea3124898843e0a7a68d99d83c9f83" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 35133 } }
I20260808 04:04:06.354698 8729 raft_consensus.cc:374] T 4755e613b45d4fdaabc4afc8b96034f5 P ac14eb089ed14bdbbd682f20cb7e9ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:04:06.354791 8729 raft_consensus.cc:729] T 4755e613b45d4fdaabc4afc8b96034f5 P ac14eb089ed14bdbbd682f20cb7e9ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac14eb089ed14bdbbd682f20cb7e9ada, State: Initialized, Role: FOLLOWER
I20260808 04:04:06.355007 8729 consensus_queue.cc:260] T 4755e613b45d4fdaabc4afc8b96034f5 P ac14eb089ed14bdbbd682f20cb7e9ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac14eb089ed14bdbbd682f20cb7e9ada" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42739 } } peers { permanent_uuid: "1f623bcbb73947e8ba2becbff3aa228f" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 41193 } } peers { permanent_uuid: "09ea3124898843e0a7a68d99d83c9f83" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 35133 } }
I20260808 04:04:06.355599 8729 ts_tablet_manager.cc:1428] T 4755e613b45d4fdaabc4afc8b96034f5 P ac14eb089ed14bdbbd682f20cb7e9ada: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260808 04:04:06.526201 8734 raft_consensus.cc:482] T 4755e613b45d4fdaabc4afc8b96034f5 P 1f623bcbb73947e8ba2becbff3aa228f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:04:06.526456 8734 raft_consensus.cc:504] T 4755e613b45d4fdaabc4afc8b96034f5 P 1f623bcbb73947e8ba2becbff3aa228f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac14eb089ed14bdbbd682f20cb7e9ada" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42739 } } peers { permanent_uuid: "1f623bcbb73947e8ba2becbff3aa228f" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 41193 } } peers { permanent_uuid: "09ea3124898843e0a7a68d99d83c9f83" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 35133 } }
I20260808 04:04:06.527079 8734 leader_election.cc:288] T 4755e613b45d4fdaabc4afc8b96034f5 P 1f623bcbb73947e8ba2becbff3aa228f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac14eb089ed14bdbbd682f20cb7e9ada (127.5.83.1:42739), 09ea3124898843e0a7a68d99d83c9f83 (127.5.83.3:35133)
I20260808 04:04:06.527652 8090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4755e613b45d4fdaabc4afc8b96034f5" candidate_uuid: "1f623bcbb73947e8ba2becbff3aa228f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac14eb089ed14bdbbd682f20cb7e9ada" is_pre_election: true
I20260808 04:04:06.527689 8461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4755e613b45d4fdaabc4afc8b96034f5" candidate_uuid: "1f623bcbb73947e8ba2becbff3aa228f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ea3124898843e0a7a68d99d83c9f83" is_pre_election: true
I20260808 04:04:06.527917 8090 raft_consensus.cc:2454] T 4755e613b45d4fdaabc4afc8b96034f5 P ac14eb089ed14bdbbd682f20cb7e9ada [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f623bcbb73947e8ba2becbff3aa228f in term 0.
I20260808 04:04:06.527943 8461 raft_consensus.cc:2454] T 4755e613b45d4fdaabc4afc8b96034f5 P 09ea3124898843e0a7a68d99d83c9f83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f623bcbb73947e8ba2becbff3aa228f in term 0.
I20260808 04:04:06.528381 8187 leader_election.cc:302] T 4755e613b45d4fdaabc4afc8b96034f5 P 1f623bcbb73947e8ba2becbff3aa228f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f623bcbb73947e8ba2becbff3aa228f, ac14eb089ed14bdbbd682f20cb7e9ada; no voters:
I20260808 04:04:06.528649 8734 raft_consensus.cc:2781] T 4755e613b45d4fdaabc4afc8b96034f5 P 1f623bcbb73947e8ba2becbff3aa228f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:04:06.528752 8734 raft_consensus.cc:482] T 4755e613b45d4fdaabc4afc8b96034f5 P 1f623bcbb73947e8ba2becbff3aa228f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:04:06.528836 8734 raft_consensus.cc:3037] T 4755e613b45d4fdaabc4afc8b96034f5 P 1f623bcbb73947e8ba2becbff3aa228f [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:04:06.531268 8734 raft_consensus.cc:504] T 4755e613b45d4fdaabc4afc8b96034f5 P 1f623bcbb73947e8ba2becbff3aa228f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac14eb089ed14bdbbd682f20cb7e9ada" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42739 } } peers { permanent_uuid: "1f623bcbb73947e8ba2becbff3aa228f" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 41193 } } peers { permanent_uuid: "09ea3124898843e0a7a68d99d83c9f83" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 35133 } }
I20260808 04:04:06.531773 8734 leader_election.cc:288] T 4755e613b45d4fdaabc4afc8b96034f5 P 1f623bcbb73947e8ba2becbff3aa228f [CANDIDATE]: Term 1 election: Requested vote from peers ac14eb089ed14bdbbd682f20cb7e9ada (127.5.83.1:42739), 09ea3124898843e0a7a68d99d83c9f83 (127.5.83.3:35133)
I20260808 04:04:06.532241 8090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4755e613b45d4fdaabc4afc8b96034f5" candidate_uuid: "1f623bcbb73947e8ba2becbff3aa228f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac14eb089ed14bdbbd682f20cb7e9ada"
I20260808 04:04:06.532359 8461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4755e613b45d4fdaabc4afc8b96034f5" candidate_uuid: "1f623bcbb73947e8ba2becbff3aa228f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ea3124898843e0a7a68d99d83c9f83"
I20260808 04:04:06.532460 8090 raft_consensus.cc:3037] T 4755e613b45d4fdaabc4afc8b96034f5 P ac14eb089ed14bdbbd682f20cb7e9ada [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:04:06.532565 8461 raft_consensus.cc:3037] T 4755e613b45d4fdaabc4afc8b96034f5 P 09ea3124898843e0a7a68d99d83c9f83 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:04:06.535444 8090 raft_consensus.cc:2454] T 4755e613b45d4fdaabc4afc8b96034f5 P ac14eb089ed14bdbbd682f20cb7e9ada [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f623bcbb73947e8ba2becbff3aa228f in term 1.
I20260808 04:04:06.535444 8461 raft_consensus.cc:2454] T 4755e613b45d4fdaabc4afc8b96034f5 P 09ea3124898843e0a7a68d99d83c9f83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f623bcbb73947e8ba2becbff3aa228f in term 1.
I20260808 04:04:06.535821 8187 leader_election.cc:302] T 4755e613b45d4fdaabc4afc8b96034f5 P 1f623bcbb73947e8ba2becbff3aa228f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ea3124898843e0a7a68d99d83c9f83, 1f623bcbb73947e8ba2becbff3aa228f; no voters:
I20260808 04:04:06.536075 8734 raft_consensus.cc:2781] T 4755e613b45d4fdaabc4afc8b96034f5 P 1f623bcbb73947e8ba2becbff3aa228f [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:04:06.536490 8734 raft_consensus.cc:686] T 4755e613b45d4fdaabc4afc8b96034f5 P 1f623bcbb73947e8ba2becbff3aa228f [term 1 LEADER]: Becoming Leader. State: Replica: 1f623bcbb73947e8ba2becbff3aa228f, State: Running, Role: LEADER
I20260808 04:04:06.536772 8734 consensus_queue.cc:237] T 4755e613b45d4fdaabc4afc8b96034f5 P 1f623bcbb73947e8ba2becbff3aa228f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac14eb089ed14bdbbd682f20cb7e9ada" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42739 } } peers { permanent_uuid: "1f623bcbb73947e8ba2becbff3aa228f" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 41193 } } peers { permanent_uuid: "09ea3124898843e0a7a68d99d83c9f83" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 35133 } }
I20260808 04:04:06.539192 7874 catalog_manager.cc:5495] T 4755e613b45d4fdaabc4afc8b96034f5 P 1f623bcbb73947e8ba2becbff3aa228f reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f623bcbb73947e8ba2becbff3aa228f (127.5.83.2). New cstate: current_term: 1 leader_uuid: "1f623bcbb73947e8ba2becbff3aa228f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac14eb089ed14bdbbd682f20cb7e9ada" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42739 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f623bcbb73947e8ba2becbff3aa228f" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 41193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09ea3124898843e0a7a68d99d83c9f83" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 35133 } health_report { overall_health: UNKNOWN } } }
I20260808 04:04:06.593403 8740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:06.588256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57580 (local address 127.5.83.2:41193)
0808 04:04:06.588895 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:06.588906 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:06.588925 (+ 19us) server_negotiation.cc:407] Connection header received
0808 04:04:06.588975 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:06.588978 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:06.589036 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:06.589119 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:04:06.589540 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:06.590323 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:06.591625 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:06.592124 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:06.592238 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:06.592671 (+ 433us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:04:06.592774 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:04:06.592850 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:06.593015 (+ 165us) server_negotiation.cc:299] Negotiation successful
0808 04:04:06.593095 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":375,"threads_started":1}
I20260808 04:04:06.602686 8461 raft_consensus.cc:1261] T 4755e613b45d4fdaabc4afc8b96034f5 P 09ea3124898843e0a7a68d99d83c9f83 [term 1 FOLLOWER]: Refusing update from remote peer 1f623bcbb73947e8ba2becbff3aa228f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:04:06.602722 8090 raft_consensus.cc:1261] T 4755e613b45d4fdaabc4afc8b96034f5 P ac14eb089ed14bdbbd682f20cb7e9ada [term 1 FOLLOWER]: Refusing update from remote peer 1f623bcbb73947e8ba2becbff3aa228f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:04:06.603235 8737 consensus_queue.cc:1035] T 4755e613b45d4fdaabc4afc8b96034f5 P 1f623bcbb73947e8ba2becbff3aa228f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac14eb089ed14bdbbd682f20cb7e9ada" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:04:06.603739 8734 consensus_queue.cc:1035] T 4755e613b45d4fdaabc4afc8b96034f5 P 1f623bcbb73947e8ba2becbff3aa228f [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ea3124898843e0a7a68d99d83c9f83" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 35133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:04:06.647610 7874 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:04:06.647868 7874 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:04:06.650496 7874 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4f45675d9036448fbf2bbd0b2d808bc7: Sending DeleteTablet for 3 replicas of tablet 4755e613b45d4fdaabc4afc8b96034f5
I20260808 04:04:06.651177 8057 tablet_service.cc:1514] Processing DeleteTablet for tablet 4755e613b45d4fdaabc4afc8b96034f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:04:06 UTC) from {username='kudu', principal='kudu/127.5.83.62@KRBTEST.COM'} at 127.0.0.1:56412
I20260808 04:04:06.651201 8246 tablet_service.cc:1514] Processing DeleteTablet for tablet 4755e613b45d4fdaabc4afc8b96034f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:04:06 UTC) from {username='kudu', principal='kudu/127.5.83.62@KRBTEST.COM'} at 127.0.0.1:57554
I20260808 04:04:06.651400 8436 tablet_service.cc:1514] Processing DeleteTablet for tablet 4755e613b45d4fdaabc4afc8b96034f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:04:06 UTC) from {username='kudu', principal='kudu/127.5.83.62@KRBTEST.COM'} at 127.0.0.1:41000
I20260808 04:04:06.651471 8705 tablet_replica.cc:330] T 4755e613b45d4fdaabc4afc8b96034f5 P ac14eb089ed14bdbbd682f20cb7e9ada: stopping tablet replica
I20260808 04:04:06.651481 8706 tablet_replica.cc:330] T 4755e613b45d4fdaabc4afc8b96034f5 P 1f623bcbb73947e8ba2becbff3aa228f: stopping tablet replica
I20260808 04:04:06.651655 8705 raft_consensus.cc:2229] T 4755e613b45d4fdaabc4afc8b96034f5 P ac14eb089ed14bdbbd682f20cb7e9ada [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:04:06.651662 8706 raft_consensus.cc:2229] T 4755e613b45d4fdaabc4afc8b96034f5 P 1f623bcbb73947e8ba2becbff3aa228f [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:04:06.651855 8705 raft_consensus.cc:2258] T 4755e613b45d4fdaabc4afc8b96034f5 P ac14eb089ed14bdbbd682f20cb7e9ada [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:04:06.651875 8708 tablet_replica.cc:330] T 4755e613b45d4fdaabc4afc8b96034f5 P 09ea3124898843e0a7a68d99d83c9f83: stopping tablet replica
I20260808 04:04:06.651974 8706 raft_consensus.cc:2258] T 4755e613b45d4fdaabc4afc8b96034f5 P 1f623bcbb73947e8ba2becbff3aa228f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:04:06.652073 8708 raft_consensus.cc:2229] T 4755e613b45d4fdaabc4afc8b96034f5 P 09ea3124898843e0a7a68d99d83c9f83 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:04:06.652240 8708 raft_consensus.cc:2258] T 4755e613b45d4fdaabc4afc8b96034f5 P 09ea3124898843e0a7a68d99d83c9f83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:04:06.652771 7874 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48922:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260808 04:04:06.653000 8706 ts_tablet_manager.cc:1905] T 4755e613b45d4fdaabc4afc8b96034f5 P 1f623bcbb73947e8ba2becbff3aa228f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:04:06.652990 8705 ts_tablet_manager.cc:1905] T 4755e613b45d4fdaabc4afc8b96034f5 P ac14eb089ed14bdbbd682f20cb7e9ada: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260808 04:04:06.653316 7874 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:04:06.654831 8708 ts_tablet_manager.cc:1905] T 4755e613b45d4fdaabc4afc8b96034f5 P 09ea3124898843e0a7a68d99d83c9f83: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:04:06.659685 8057 tablet_service.cc:1467] Processing CreateTablet for tablet f8a24d4fbfc148588c8b8cc8bcc0b4d0 (DEFAULT_TABLE table=test-table [id=23b36690ee9f4329b9d2dc98e2e45847]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:04:06.659849 8246 tablet_service.cc:1467] Processing CreateTablet for tablet f8a24d4fbfc148588c8b8cc8bcc0b4d0 (DEFAULT_TABLE table=test-table [id=23b36690ee9f4329b9d2dc98e2e45847]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:04:06.660210 8246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8a24d4fbfc148588c8b8cc8bcc0b4d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:04:06.660346 8057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8a24d4fbfc148588c8b8cc8bcc0b4d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:04:06.660876 8708 ts_tablet_manager.cc:1918] T 4755e613b45d4fdaabc4afc8b96034f5 P 09ea3124898843e0a7a68d99d83c9f83: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 04:04:06.661026 8708 log.cc:1196] T 4755e613b45d4fdaabc4afc8b96034f5 P 09ea3124898843e0a7a68d99d83c9f83: Deleting WAL directory at /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/minicluster-data/ts-2/wal/wals/4755e613b45d4fdaabc4afc8b96034f5
I20260808 04:04:06.661427 8708 ts_tablet_manager.cc:1939] T 4755e613b45d4fdaabc4afc8b96034f5 P 09ea3124898843e0a7a68d99d83c9f83: Deleting consensus metadata
I20260808 04:04:06.666293 8729 tablet_bootstrap.cc:492] T f8a24d4fbfc148588c8b8cc8bcc0b4d0 P ac14eb089ed14bdbbd682f20cb7e9ada: Bootstrap starting.
I20260808 04:04:06.660337 8705 ts_tablet_manager.cc:1918] T 4755e613b45d4fdaabc4afc8b96034f5 P ac14eb089ed14bdbbd682f20cb7e9ada: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 04:04:06.666373 8728 tablet_bootstrap.cc:492] T f8a24d4fbfc148588c8b8cc8bcc0b4d0 P 1f623bcbb73947e8ba2becbff3aa228f: Bootstrap starting.
I20260808 04:04:06.666512 8705 log.cc:1196] T 4755e613b45d4fdaabc4afc8b96034f5 P ac14eb089ed14bdbbd682f20cb7e9ada: Deleting WAL directory at /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/minicluster-data/ts-0/wal/wals/4755e613b45d4fdaabc4afc8b96034f5
I20260808 04:04:06.666963 8705 ts_tablet_manager.cc:1939] T 4755e613b45d4fdaabc4afc8b96034f5 P ac14eb089ed14bdbbd682f20cb7e9ada: Deleting consensus metadata
I20260808 04:04:06.667012 7859 catalog_manager.cc:4841] TS 09ea3124898843e0a7a68d99d83c9f83 (127.5.83.3:35133): tablet 4755e613b45d4fdaabc4afc8b96034f5 (table test-table [id=3a6a8887f89d4b328d11f53d80ac8167]) successfully deleted
I20260808 04:04:06.667659 8436 tablet_service.cc:1467] Processing CreateTablet for tablet f8a24d4fbfc148588c8b8cc8bcc0b4d0 (DEFAULT_TABLE table=test-table [id=23b36690ee9f4329b9d2dc98e2e45847]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:04:06.667865 8728 tablet_bootstrap.cc:654] T f8a24d4fbfc148588c8b8cc8bcc0b4d0 P 1f623bcbb73947e8ba2becbff3aa228f: Neither blocks nor log segments found. Creating new log.
I20260808 04:04:06.667935 8729 tablet_bootstrap.cc:654] T f8a24d4fbfc148588c8b8cc8bcc0b4d0 P ac14eb089ed14bdbbd682f20cb7e9ada: Neither blocks nor log segments found. Creating new log.
I20260808 04:04:06.668100 8436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8a24d4fbfc148588c8b8cc8bcc0b4d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:04:06.668098 7859 catalog_manager.cc:4841] TS ac14eb089ed14bdbbd682f20cb7e9ada (127.5.83.1:42739): tablet 4755e613b45d4fdaabc4afc8b96034f5 (table test-table [id=3a6a8887f89d4b328d11f53d80ac8167]) successfully deleted
I20260808 04:04:06.668990 8706 ts_tablet_manager.cc:1918] T 4755e613b45d4fdaabc4afc8b96034f5 P 1f623bcbb73947e8ba2becbff3aa228f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 04:04:06.669131 8706 log.cc:1196] T 4755e613b45d4fdaabc4afc8b96034f5 P 1f623bcbb73947e8ba2becbff3aa228f: Deleting WAL directory at /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/minicluster-data/ts-1/wal/wals/4755e613b45d4fdaabc4afc8b96034f5
I20260808 04:04:06.669556 8706 ts_tablet_manager.cc:1939] T 4755e613b45d4fdaabc4afc8b96034f5 P 1f623bcbb73947e8ba2becbff3aa228f: Deleting consensus metadata
I20260808 04:04:06.670825 7859 catalog_manager.cc:4841] TS 1f623bcbb73947e8ba2becbff3aa228f (127.5.83.2:41193): tablet 4755e613b45d4fdaabc4afc8b96034f5 (table test-table [id=3a6a8887f89d4b328d11f53d80ac8167]) successfully deleted
I20260808 04:04:06.672258 8728 tablet_bootstrap.cc:492] T f8a24d4fbfc148588c8b8cc8bcc0b4d0 P 1f623bcbb73947e8ba2becbff3aa228f: No bootstrap required, opened a new log
I20260808 04:04:06.672436 8728 ts_tablet_manager.cc:1397] T f8a24d4fbfc148588c8b8cc8bcc0b4d0 P 1f623bcbb73947e8ba2becbff3aa228f: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260808 04:04:06.673032 8729 tablet_bootstrap.cc:492] T f8a24d4fbfc148588c8b8cc8bcc0b4d0 P ac14eb089ed14bdbbd682f20cb7e9ada: No bootstrap required, opened a new log
I20260808 04:04:06.673194 8729 ts_tablet_manager.cc:1397] T f8a24d4fbfc148588c8b8cc8bcc0b4d0 P ac14eb089ed14bdbbd682f20cb7e9ada: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260808 04:04:06.673146 8728 raft_consensus.cc:348] T f8a24d4fbfc148588c8b8cc8bcc0b4d0 P 1f623bcbb73947e8ba2becbff3aa228f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac14eb089ed14bdbbd682f20cb7e9ada" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42739 } } peers { permanent_uuid: "1f623bcbb73947e8ba2becbff3aa228f" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 41193 } } peers { permanent_uuid: "09ea3124898843e0a7a68d99d83c9f83" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 35133 } }
I20260808 04:04:06.673403 8728 raft_consensus.cc:374] T f8a24d4fbfc148588c8b8cc8bcc0b4d0 P 1f623bcbb73947e8ba2becbff3aa228f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:04:06.673486 8728 raft_consensus.cc:729] T f8a24d4fbfc148588c8b8cc8bcc0b4d0 P 1f623bcbb73947e8ba2becbff3aa228f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f623bcbb73947e8ba2becbff3aa228f, State: Initialized, Role: FOLLOWER
I20260808 04:04:06.673715 8728 consensus_queue.cc:260] T f8a24d4fbfc148588c8b8cc8bcc0b4d0 P 1f623bcbb73947e8ba2becbff3aa228f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac14eb089ed14bdbbd682f20cb7e9ada" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42739 } } peers { permanent_uuid: "1f623bcbb73947e8ba2becbff3aa228f" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 41193 } } peers { permanent_uuid: "09ea3124898843e0a7a68d99d83c9f83" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 35133 } }
I20260808 04:04:06.674043 8727 tablet_bootstrap.cc:492] T f8a24d4fbfc148588c8b8cc8bcc0b4d0 P 09ea3124898843e0a7a68d99d83c9f83: Bootstrap starting.
I20260808 04:04:06.674216 8728 ts_tablet_manager.cc:1428] T f8a24d4fbfc148588c8b8cc8bcc0b4d0 P 1f623bcbb73947e8ba2becbff3aa228f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260808 04:04:06.673929 8729 raft_consensus.cc:348] T f8a24d4fbfc148588c8b8cc8bcc0b4d0 P ac14eb089ed14bdbbd682f20cb7e9ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac14eb089ed14bdbbd682f20cb7e9ada" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42739 } } peers { permanent_uuid: "1f623bcbb73947e8ba2becbff3aa228f" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 41193 } } peers { permanent_uuid: "09ea3124898843e0a7a68d99d83c9f83" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 35133 } }
I20260808 04:04:06.674350 8729 raft_consensus.cc:374] T f8a24d4fbfc148588c8b8cc8bcc0b4d0 P ac14eb089ed14bdbbd682f20cb7e9ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:04:06.674436 8729 raft_consensus.cc:729] T f8a24d4fbfc148588c8b8cc8bcc0b4d0 P ac14eb089ed14bdbbd682f20cb7e9ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac14eb089ed14bdbbd682f20cb7e9ada, State: Initialized, Role: FOLLOWER
I20260808 04:04:06.674656 8729 consensus_queue.cc:260] T f8a24d4fbfc148588c8b8cc8bcc0b4d0 P ac14eb089ed14bdbbd682f20cb7e9ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac14eb089ed14bdbbd682f20cb7e9ada" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42739 } } peers { permanent_uuid: "1f623bcbb73947e8ba2becbff3aa228f" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 41193 } } peers { permanent_uuid: "09ea3124898843e0a7a68d99d83c9f83" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 35133 } }
I20260808 04:04:06.675113 8729 ts_tablet_manager.cc:1428] T f8a24d4fbfc148588c8b8cc8bcc0b4d0 P ac14eb089ed14bdbbd682f20cb7e9ada: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260808 04:04:06.675433 8727 tablet_bootstrap.cc:654] T f8a24d4fbfc148588c8b8cc8bcc0b4d0 P 09ea3124898843e0a7a68d99d83c9f83: Neither blocks nor log segments found. Creating new log.
I20260808 04:04:06.679375 8727 tablet_bootstrap.cc:492] T f8a24d4fbfc148588c8b8cc8bcc0b4d0 P 09ea3124898843e0a7a68d99d83c9f83: No bootstrap required, opened a new log
I20260808 04:04:06.679533 8727 ts_tablet_manager.cc:1397] T f8a24d4fbfc148588c8b8cc8bcc0b4d0 P 09ea3124898843e0a7a68d99d83c9f83: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260808 04:04:06.680259 8727 raft_consensus.cc:348] T f8a24d4fbfc148588c8b8cc8bcc0b4d0 P 09ea3124898843e0a7a68d99d83c9f83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac14eb089ed14bdbbd682f20cb7e9ada" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42739 } } peers { permanent_uuid: "1f623bcbb73947e8ba2becbff3aa228f" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 41193 } } peers { permanent_uuid: "09ea3124898843e0a7a68d99d83c9f83" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 35133 } }
I20260808 04:04:06.680493 8727 raft_consensus.cc:374] T f8a24d4fbfc148588c8b8cc8bcc0b4d0 P 09ea3124898843e0a7a68d99d83c9f83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:04:06.680567 8727 raft_consensus.cc:729] T f8a24d4fbfc148588c8b8cc8bcc0b4d0 P 09ea3124898843e0a7a68d99d83c9f83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ea3124898843e0a7a68d99d83c9f83, State: Initialized, Role: FOLLOWER
I20260808 04:04:06.680778 8727 consensus_queue.cc:260] T f8a24d4fbfc148588c8b8cc8bcc0b4d0 P 09ea3124898843e0a7a68d99d83c9f83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac14eb089ed14bdbbd682f20cb7e9ada" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42739 } } peers { permanent_uuid: "1f623bcbb73947e8ba2becbff3aa228f" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 41193 } } peers { permanent_uuid: "09ea3124898843e0a7a68d99d83c9f83" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 35133 } }
I20260808 04:04:06.681277 8727 ts_tablet_manager.cc:1428] T f8a24d4fbfc148588c8b8cc8bcc0b4d0 P 09ea3124898843e0a7a68d99d83c9f83: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260808 04:04:06.839505 8690 raft_consensus.cc:482] T f8a24d4fbfc148588c8b8cc8bcc0b4d0 P 09ea3124898843e0a7a68d99d83c9f83 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:04:06.839751 8690 raft_consensus.cc:504] T f8a24d4fbfc148588c8b8cc8bcc0b4d0 P 09ea3124898843e0a7a68d99d83c9f83 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac14eb089ed14bdbbd682f20cb7e9ada" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42739 } } peers { permanent_uuid: "1f623bcbb73947e8ba2becbff3aa228f" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 41193 } } peers { permanent_uuid: "09ea3124898843e0a7a68d99d83c9f83" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 35133 } }
I20260808 04:04:06.840389 8690 leader_election.cc:288] T f8a24d4fbfc148588c8b8cc8bcc0b4d0 P 09ea3124898843e0a7a68d99d83c9f83 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac14eb089ed14bdbbd682f20cb7e9ada (127.5.83.1:42739), 1f623bcbb73947e8ba2becbff3aa228f (127.5.83.2:41193)
I20260808 04:04:06.840991 8090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8a24d4fbfc148588c8b8cc8bcc0b4d0" candidate_uuid: "09ea3124898843e0a7a68d99d83c9f83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac14eb089ed14bdbbd682f20cb7e9ada" is_pre_election: true
I20260808 04:04:06.841043 8281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8a24d4fbfc148588c8b8cc8bcc0b4d0" candidate_uuid: "09ea3124898843e0a7a68d99d83c9f83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f623bcbb73947e8ba2becbff3aa228f" is_pre_election: true
I20260808 04:04:06.841236 8090 raft_consensus.cc:2454] T f8a24d4fbfc148588c8b8cc8bcc0b4d0 P ac14eb089ed14bdbbd682f20cb7e9ada [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09ea3124898843e0a7a68d99d83c9f83 in term 0.
I20260808 04:04:06.841372 8281 raft_consensus.cc:2454] T f8a24d4fbfc148588c8b8cc8bcc0b4d0 P 1f623bcbb73947e8ba2becbff3aa228f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09ea3124898843e0a7a68d99d83c9f83 in term 0.
I20260808 04:04:06.841729 8375 leader_election.cc:302] T f8a24d4fbfc148588c8b8cc8bcc0b4d0 P 09ea3124898843e0a7a68d99d83c9f83 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ea3124898843e0a7a68d99d83c9f83, ac14eb089ed14bdbbd682f20cb7e9ada; no voters:
I20260808 04:04:06.842029 8690 raft_consensus.cc:2781] T f8a24d4fbfc148588c8b8cc8bcc0b4d0 P 09ea3124898843e0a7a68d99d83c9f83 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:04:06.842142 8690 raft_consensus.cc:482] T f8a24d4fbfc148588c8b8cc8bcc0b4d0 P 09ea3124898843e0a7a68d99d83c9f83 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:04:06.842233 8690 raft_consensus.cc:3037] T f8a24d4fbfc148588c8b8cc8bcc0b4d0 P 09ea3124898843e0a7a68d99d83c9f83 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:04:06.844597 8690 raft_consensus.cc:504] T f8a24d4fbfc148588c8b8cc8bcc0b4d0 P 09ea3124898843e0a7a68d99d83c9f83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac14eb089ed14bdbbd682f20cb7e9ada" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42739 } } peers { permanent_uuid: "1f623bcbb73947e8ba2becbff3aa228f" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 41193 } } peers { permanent_uuid: "09ea3124898843e0a7a68d99d83c9f83" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 35133 } }
I20260808 04:04:06.845067 8690 leader_election.cc:288] T f8a24d4fbfc148588c8b8cc8bcc0b4d0 P 09ea3124898843e0a7a68d99d83c9f83 [CANDIDATE]: Term 1 election: Requested vote from peers ac14eb089ed14bdbbd682f20cb7e9ada (127.5.83.1:42739), 1f623bcbb73947e8ba2becbff3aa228f (127.5.83.2:41193)
I20260808 04:04:06.845525 8090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8a24d4fbfc148588c8b8cc8bcc0b4d0" candidate_uuid: "09ea3124898843e0a7a68d99d83c9f83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac14eb089ed14bdbbd682f20cb7e9ada"
I20260808 04:04:06.845623 8281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8a24d4fbfc148588c8b8cc8bcc0b4d0" candidate_uuid: "09ea3124898843e0a7a68d99d83c9f83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f623bcbb73947e8ba2becbff3aa228f"
I20260808 04:04:06.845695 8090 raft_consensus.cc:3037] T f8a24d4fbfc148588c8b8cc8bcc0b4d0 P ac14eb089ed14bdbbd682f20cb7e9ada [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:04:06.845803 8281 raft_consensus.cc:3037] T f8a24d4fbfc148588c8b8cc8bcc0b4d0 P 1f623bcbb73947e8ba2becbff3aa228f [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:04:06.847697 8090 raft_consensus.cc:2454] T f8a24d4fbfc148588c8b8cc8bcc0b4d0 P ac14eb089ed14bdbbd682f20cb7e9ada [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ea3124898843e0a7a68d99d83c9f83 in term 1.
I20260808 04:04:06.847822 8281 raft_consensus.cc:2454] T f8a24d4fbfc148588c8b8cc8bcc0b4d0 P 1f623bcbb73947e8ba2becbff3aa228f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ea3124898843e0a7a68d99d83c9f83 in term 1.
I20260808 04:04:06.848129 8375 leader_election.cc:302] T f8a24d4fbfc148588c8b8cc8bcc0b4d0 P 09ea3124898843e0a7a68d99d83c9f83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ea3124898843e0a7a68d99d83c9f83, ac14eb089ed14bdbbd682f20cb7e9ada; no voters:
I20260808 04:04:06.848387 8690 raft_consensus.cc:2781] T f8a24d4fbfc148588c8b8cc8bcc0b4d0 P 09ea3124898843e0a7a68d99d83c9f83 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:04:06.848739 8690 raft_consensus.cc:686] T f8a24d4fbfc148588c8b8cc8bcc0b4d0 P 09ea3124898843e0a7a68d99d83c9f83 [term 1 LEADER]: Becoming Leader. State: Replica: 09ea3124898843e0a7a68d99d83c9f83, State: Running, Role: LEADER
I20260808 04:04:06.848994 8690 consensus_queue.cc:237] T f8a24d4fbfc148588c8b8cc8bcc0b4d0 P 09ea3124898843e0a7a68d99d83c9f83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac14eb089ed14bdbbd682f20cb7e9ada" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42739 } } peers { permanent_uuid: "1f623bcbb73947e8ba2becbff3aa228f" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 41193 } } peers { permanent_uuid: "09ea3124898843e0a7a68d99d83c9f83" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 35133 } }
I20260808 04:04:06.851508 7873 catalog_manager.cc:5495] T f8a24d4fbfc148588c8b8cc8bcc0b4d0 P 09ea3124898843e0a7a68d99d83c9f83 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09ea3124898843e0a7a68d99d83c9f83 (127.5.83.3). New cstate: current_term: 1 leader_uuid: "09ea3124898843e0a7a68d99d83c9f83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac14eb089ed14bdbbd682f20cb7e9ada" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42739 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f623bcbb73947e8ba2becbff3aa228f" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 41193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09ea3124898843e0a7a68d99d83c9f83" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 35133 } health_report { overall_health: HEALTHY } } }
I20260808 04:04:06.880811 8755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:06.876209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41022 (local address 127.5.83.3:35133)
0808 04:04:06.876727 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:06.876731 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:06.876753 (+ 22us) server_negotiation.cc:407] Connection header received
0808 04:04:06.876800 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:06.876803 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:06.876860 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:06.876943 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:04:06.877316 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:06.878091 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:06.879309 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:06.879742 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:06.879838 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:06.880193 (+ 355us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:04:06.880286 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:04:06.880348 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:06.880449 (+ 101us) server_negotiation.cc:299] Negotiation successful
0808 04:04:06.880507 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":288,"threads_started":1}
I20260808 04:04:06.888732 8281 raft_consensus.cc:1261] T f8a24d4fbfc148588c8b8cc8bcc0b4d0 P 1f623bcbb73947e8ba2becbff3aa228f [term 1 FOLLOWER]: Refusing update from remote peer 09ea3124898843e0a7a68d99d83c9f83: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:04:06.888939 8090 raft_consensus.cc:1261] T f8a24d4fbfc148588c8b8cc8bcc0b4d0 P ac14eb089ed14bdbbd682f20cb7e9ada [term 1 FOLLOWER]: Refusing update from remote peer 09ea3124898843e0a7a68d99d83c9f83: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:04:06.889345 8752 consensus_queue.cc:1035] T f8a24d4fbfc148588c8b8cc8bcc0b4d0 P 09ea3124898843e0a7a68d99d83c9f83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f623bcbb73947e8ba2becbff3aa228f" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 41193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:04:06.889642 8690 consensus_queue.cc:1035] T f8a24d4fbfc148588c8b8cc8bcc0b4d0 P 09ea3124898843e0a7a68d99d83c9f83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac14eb089ed14bdbbd682f20cb7e9ada" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 42739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 04:04:06.909417 8676 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260808 04:04:08.483155 7874 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:04:08.483389 7874 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:04:08.487157 7874 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4f45675d9036448fbf2bbd0b2d808bc7: Sending DeleteTablet for 3 replicas of tablet f8a24d4fbfc148588c8b8cc8bcc0b4d0
I20260808 04:04:08.487808 8057 tablet_service.cc:1514] Processing DeleteTablet for tablet f8a24d4fbfc148588c8b8cc8bcc0b4d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:04:08 UTC) from {username='kudu', principal='kudu/127.5.83.62@KRBTEST.COM'} at 127.0.0.1:56412
I20260808 04:04:08.487946 8246 tablet_service.cc:1514] Processing DeleteTablet for tablet f8a24d4fbfc148588c8b8cc8bcc0b4d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:04:08 UTC) from {username='kudu', principal='kudu/127.5.83.62@KRBTEST.COM'} at 127.0.0.1:57554
I20260808 04:04:08.487972 8436 tablet_service.cc:1514] Processing DeleteTablet for tablet f8a24d4fbfc148588c8b8cc8bcc0b4d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:04:08 UTC) from {username='kudu', principal='kudu/127.5.83.62@KRBTEST.COM'} at 127.0.0.1:41000
I20260808 04:04:08.488942 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 7969
I20260808 04:04:08.500464 8798 tablet_replica.cc:330] T f8a24d4fbfc148588c8b8cc8bcc0b4d0 P 09ea3124898843e0a7a68d99d83c9f83: stopping tablet replica
I20260808 04:04:08.500664 8798 raft_consensus.cc:2229] T f8a24d4fbfc148588c8b8cc8bcc0b4d0 P 09ea3124898843e0a7a68d99d83c9f83 [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:04:08.501021 8798 raft_consensus.cc:2258] T f8a24d4fbfc148588c8b8cc8bcc0b4d0 P 09ea3124898843e0a7a68d99d83c9f83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:04:08.501719 8795 tablet_replica.cc:330] T f8a24d4fbfc148588c8b8cc8bcc0b4d0 P ac14eb089ed14bdbbd682f20cb7e9ada: stopping tablet replica
I20260808 04:04:08.501920 8795 raft_consensus.cc:2229] T f8a24d4fbfc148588c8b8cc8bcc0b4d0 P ac14eb089ed14bdbbd682f20cb7e9ada [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:04:08.502162 8795 raft_consensus.cc:2258] T f8a24d4fbfc148588c8b8cc8bcc0b4d0 P ac14eb089ed14bdbbd682f20cb7e9ada [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:04:08.502673 8798 ts_tablet_manager.cc:1905] T f8a24d4fbfc148588c8b8cc8bcc0b4d0 P 09ea3124898843e0a7a68d99d83c9f83: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:04:08.503152 8795 ts_tablet_manager.cc:1905] T f8a24d4fbfc148588c8b8cc8bcc0b4d0 P ac14eb089ed14bdbbd682f20cb7e9ada: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:04:08.510113 8795 ts_tablet_manager.cc:1918] T f8a24d4fbfc148588c8b8cc8bcc0b4d0 P ac14eb089ed14bdbbd682f20cb7e9ada: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 04:04:08.510140 8798 ts_tablet_manager.cc:1918] T f8a24d4fbfc148588c8b8cc8bcc0b4d0 P 09ea3124898843e0a7a68d99d83c9f83: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 04:04:08.510277 8795 log.cc:1196] T f8a24d4fbfc148588c8b8cc8bcc0b4d0 P ac14eb089ed14bdbbd682f20cb7e9ada: Deleting WAL directory at /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/minicluster-data/ts-0/wal/wals/f8a24d4fbfc148588c8b8cc8bcc0b4d0
I20260808 04:04:08.510442 8798 log.cc:1196] T f8a24d4fbfc148588c8b8cc8bcc0b4d0 P 09ea3124898843e0a7a68d99d83c9f83: Deleting WAL directory at /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/minicluster-data/ts-2/wal/wals/f8a24d4fbfc148588c8b8cc8bcc0b4d0
I20260808 04:04:08.510787 8795 ts_tablet_manager.cc:1939] T f8a24d4fbfc148588c8b8cc8bcc0b4d0 P ac14eb089ed14bdbbd682f20cb7e9ada: Deleting consensus metadata
I20260808 04:04:08.511005 8798 ts_tablet_manager.cc:1939] T f8a24d4fbfc148588c8b8cc8bcc0b4d0 P 09ea3124898843e0a7a68d99d83c9f83: Deleting consensus metadata
I20260808 04:04:08.511945 7859 catalog_manager.cc:4841] TS ac14eb089ed14bdbbd682f20cb7e9ada (127.5.83.1:42739): tablet f8a24d4fbfc148588c8b8cc8bcc0b4d0 (table test-table [id=23b36690ee9f4329b9d2dc98e2e45847]) successfully deleted
I20260808 04:04:08.512440 7859 catalog_manager.cc:4841] TS 09ea3124898843e0a7a68d99d83c9f83 (127.5.83.3:35133): tablet f8a24d4fbfc148588c8b8cc8bcc0b4d0 (table test-table [id=23b36690ee9f4329b9d2dc98e2e45847]) successfully deleted
I20260808 04:04:08.513083 8796 tablet_replica.cc:330] T f8a24d4fbfc148588c8b8cc8bcc0b4d0 P 1f623bcbb73947e8ba2becbff3aa228f: stopping tablet replica
I20260808 04:04:08.513294 8796 raft_consensus.cc:2229] T f8a24d4fbfc148588c8b8cc8bcc0b4d0 P 1f623bcbb73947e8ba2becbff3aa228f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:04:08.513528 8796 raft_consensus.cc:2258] T f8a24d4fbfc148588c8b8cc8bcc0b4d0 P 1f623bcbb73947e8ba2becbff3aa228f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:04:08.514367 8796 ts_tablet_manager.cc:1905] T f8a24d4fbfc148588c8b8cc8bcc0b4d0 P 1f623bcbb73947e8ba2becbff3aa228f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:04:08.519466 8796 ts_tablet_manager.cc:1918] T f8a24d4fbfc148588c8b8cc8bcc0b4d0 P 1f623bcbb73947e8ba2becbff3aa228f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 04:04:08.519579 8796 log.cc:1196] T f8a24d4fbfc148588c8b8cc8bcc0b4d0 P 1f623bcbb73947e8ba2becbff3aa228f: Deleting WAL directory at /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161826605655-5452-0/minicluster-data/ts-1/wal/wals/f8a24d4fbfc148588c8b8cc8bcc0b4d0
I20260808 04:04:08.519964 8796 ts_tablet_manager.cc:1939] T f8a24d4fbfc148588c8b8cc8bcc0b4d0 P 1f623bcbb73947e8ba2becbff3aa228f: Deleting consensus metadata
I20260808 04:04:08.520745 7859 catalog_manager.cc:4841] TS 1f623bcbb73947e8ba2becbff3aa228f (127.5.83.2:41193): tablet f8a24d4fbfc148588c8b8cc8bcc0b4d0 (table test-table [id=23b36690ee9f4329b9d2dc98e2e45847]) successfully deleted
Aug 08 04:04:08 dist-test-slave-1l3l krb5kdc[7810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161846, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.83.1@KRBTEST.COM
I20260808 04:04:08.529716 8801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:08.490814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42532 (local address 127.5.83.1:42739)
0808 04:04:08.512272 (+ 21458us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:08.512278 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:08.512654 (+ 376us) server_negotiation.cc:407] Connection header received
0808 04:04:08.512892 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:08.512896 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:08.512977 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:08.513101 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:08.514386 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:08.515472 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:08.516698 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:08.517284 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:08.517431 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:08.524199 (+ 6768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:08.524214 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:08.524229 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:08.524261 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:08.527257 (+ 2996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:08.527980 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:08.527988 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:08.527990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:08.528114 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:08.528368 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:08.528371 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:08.528373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:08.528796 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:08.529046 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:08.529223 (+ 177us) server_negotiation.cc:299] Negotiation successful
0808 04:04:08.529379 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21318,"thread_start_us":21490,"threads_started":1}
W20260808 04:04:08.530334 8154 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42532
W20260808 04:04:08.530867 5452 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 7969: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 04:04:08.531064 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 7969
I20260808 04:04:08.559338 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 8164
Aug 08 04:04:08 dist-test-slave-1l3l krb5kdc[7810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161846, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.83.2@KRBTEST.COM
I20260808 04:04:08.573196 8803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:08.560013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34322 (local address 127.5.83.2:41193)
0808 04:04:08.560481 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:08.560485 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:08.560502 (+ 17us) server_negotiation.cc:407] Connection header received
0808 04:04:08.560555 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:08.560559 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:08.560622 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:08.560707 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:08.561849 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:08.562597 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:08.563856 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:08.564253 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:08.564342 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:08.568158 (+ 3816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:08.568171 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:08.568186 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:08.568226 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:08.570883 (+ 2657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:08.571464 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:08.571470 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:08.571473 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:08.571601 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:08.571925 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:08.571928 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:08.571930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:08.572284 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:08.572509 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:08.572769 (+ 260us) server_negotiation.cc:299] Negotiation successful
0808 04:04:08.572930 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":235,"threads_started":1}
W20260808 04:04:08.573830 8337 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34322
W20260808 04:04:08.574373 5452 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 8164: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 04:04:08.574524 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 8164
I20260808 04:04:08.597375 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 8349
Aug 08 04:04:08 dist-test-slave-1l3l krb5kdc[7810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161846, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.83.3@KRBTEST.COM
I20260808 04:04:08.613159 8804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:08.598111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32788 (local address 127.5.83.3:35133)
0808 04:04:08.598605 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:08.598609 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:08.598625 (+ 16us) server_negotiation.cc:407] Connection header received
0808 04:04:08.598675 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:08.598679 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:08.598735 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:08.598822 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:08.600448 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:08.601189 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:08.602894 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:08.603261 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:08.603358 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:08.608197 (+ 4839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:08.608212 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:08.608228 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:08.608259 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:08.610947 (+ 2688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:08.611481 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:08.611487 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:08.611489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:08.611605 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:08.611911 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:08.611915 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:08.611917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:08.612260 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:08.612494 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:08.612665 (+ 171us) server_negotiation.cc:299] Negotiation successful
0808 04:04:08.612829 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":242,"threads_started":1}
W20260808 04:04:08.613776 8530 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32788
W20260808 04:04:08.614354 5452 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 8349: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 04:04:08.614508 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 8349
I20260808 04:04:08.638708 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 7834
I20260808 04:04:08.639886 7927 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:04:08.754595 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 7834
2026-08-08T04:04:08Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8941 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:04:08 dist-test-slave-1l3l krb5kdc[8810](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:04:08 dist-test-slave-1l3l krb5kdc[8810](info): set up 2 sockets
Aug 08 04:04:08 dist-test-slave-1l3l krb5kdc[8810](info): commencing operation
krb5kdc: starting...
W20260808 04:04:09.919133 5452 mini_kdc.cc:121] Time spent starting KDC: real 1.117s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/krb5kdc/test-user.keytab.
Aug 08 04:04:10 dist-test-slave-1l3l krb5kdc[8810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161850, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:04:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:04:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.83.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab.
Entry for principal kudu/127.5.83.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab.
I20260808 04:04:10.100396 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:34909
--webserver_interface=127.5.83.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab
--principal=kudu/127.5.83.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:43511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.62:34909
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:04:10.407609 8826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:04:10.407908 8826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:04:10.408041 8826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:04:10.415036 8826 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:04:10.415127 8826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:04:10.415199 8826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:04:10.415272 8826 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:04:10.415350 8826 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:04:10.424604 8826 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:43511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.62:34909
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab
--principal=kudu/127.5.83.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:34909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:04:10.425762 8826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:04:10.426952 8826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:04:10.438015 8836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:10.438081 8834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:10.438099 8833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:04:10.438756 8826 server_base.cc:1034] running on GCE node
I20260808 04:04:10.439486 8826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:04:10.440827 8826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:04:10.442025 8826 hybrid_clock.cc:648] HybridClock initialized: now 1786161850441979 us; error 31 us; skew 500 ppm
I20260808 04:04:10.442334 8826 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:04:10 dist-test-slave-1l3l krb5kdc[8810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161850, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:04:10.445318 8826 init.cc:377] Logged in from keytab as kudu/127.5.83.62@KRBTEST.COM (short username kudu)
I20260808 04:04:10.446653 8826 webserver.cc:460] Webserver started at http://127.5.83.62:39027/ using document root <none> and password file <none>
I20260808 04:04:10.447279 8826 fs_manager.cc:359] Metadata directory not provided
I20260808 04:04:10.447387 8826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:04:10.447695 8826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:04:10.450560 8826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/minicluster-data/master-0/data/instance:
uuid: "aeed5c2902d6455c9436c2ecab8ae4f9"
format_stamp: "Formatted at 2026-08-08 04:04:10 on dist-test-slave-1l3l"
server_key: "96fb27b4a22390f69b2fae5a396660c0"
server_key_iv: "3652acf225a55d3268cadc6d1c6557f7"
server_key_version: "encryptionkey@0"
I20260808 04:04:10.451306 8826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/minicluster-data/master-0/wal/instance:
uuid: "aeed5c2902d6455c9436c2ecab8ae4f9"
format_stamp: "Formatted at 2026-08-08 04:04:10 on dist-test-slave-1l3l"
server_key: "96fb27b4a22390f69b2fae5a396660c0"
server_key_iv: "3652acf225a55d3268cadc6d1c6557f7"
server_key_version: "encryptionkey@0"
I20260808 04:04:10.456137 8826 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.002s
I20260808 04:04:10.459242 8846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:10.460480 8826 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:04:10.460639 8826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/minicluster-data/master-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/minicluster-data/master-0/wal
uuid: "aeed5c2902d6455c9436c2ecab8ae4f9"
format_stamp: "Formatted at 2026-08-08 04:04:10 on dist-test-slave-1l3l"
server_key: "96fb27b4a22390f69b2fae5a396660c0"
server_key_iv: "3652acf225a55d3268cadc6d1c6557f7"
server_key_version: "encryptionkey@0"
I20260808 04:04:10.460811 8826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:04:10.488693 8826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:04:10.492446 8826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:04:10.492691 8826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:04:10.509454 8826 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.62:34909
I20260808 04:04:10.509521 8927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.62:34909 every 8 connection(s)
I20260808 04:04:10.510712 8826 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
I20260808 04:04:10.514305 8928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:04:10.515734 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 8826
I20260808 04:04:10.515992 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/minicluster-data/master-0/wal/instance
I20260808 04:04:10.516803 5452 external_mini_cluster.cc:1428] Setting key bcd10d9e8809badcb1058470134c4aea
I20260808 04:04:10.527194 8928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aeed5c2902d6455c9436c2ecab8ae4f9: Bootstrap starting.
Aug 08 04:04:10 dist-test-slave-1l3l krb5kdc[8810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161850, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:04:10.530364 8928 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aeed5c2902d6455c9436c2ecab8ae4f9: Neither blocks nor log segments found. Creating new log.
I20260808 04:04:10.531333 8928 log.cc:824] T 00000000000000000000000000000000 P aeed5c2902d6455c9436c2ecab8ae4f9: Log is configured to *not* fsync() on all Append() calls
I20260808 04:04:10.534086 8928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aeed5c2902d6455c9436c2ecab8ae4f9: No bootstrap required, opened a new log
I20260808 04:04:10.538085 8932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:10.518794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39186 (local address 127.5.83.62:34909)
0808 04:04:10.519653 (+ 859us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:10.519670 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:10.519748 (+ 78us) server_negotiation.cc:407] Connection header received
0808 04:04:10.521499 (+ 1751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:10.521520 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:10.521870 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:10.522243 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:10.523674 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:10.525112 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:10.526364 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:10.527028 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:10.527190 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:10.531205 (+ 4015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:10.531215 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:10.531229 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:10.531266 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:10.534230 (+ 2964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:10.534855 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:10.534861 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:10.534867 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:10.535035 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:10.535419 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:10.535422 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:10.535425 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:10.535996 (+ 571us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:10.536295 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:10.536746 (+ 451us) server_negotiation.cc:299] Negotiation successful
0808 04:04:10.537147 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":345,"threads_started":1}
I20260808 04:04:10.542224 8928 raft_consensus.cc:348] T 00000000000000000000000000000000 P aeed5c2902d6455c9436c2ecab8ae4f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeed5c2902d6455c9436c2ecab8ae4f9" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 34909 } }
I20260808 04:04:10.542548 8928 raft_consensus.cc:374] T 00000000000000000000000000000000 P aeed5c2902d6455c9436c2ecab8ae4f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:04:10.542639 8928 raft_consensus.cc:729] T 00000000000000000000000000000000 P aeed5c2902d6455c9436c2ecab8ae4f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aeed5c2902d6455c9436c2ecab8ae4f9, State: Initialized, Role: FOLLOWER
I20260808 04:04:10.543164 8928 consensus_queue.cc:260] T 00000000000000000000000000000000 P aeed5c2902d6455c9436c2ecab8ae4f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeed5c2902d6455c9436c2ecab8ae4f9" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 34909 } }
I20260808 04:04:10.543413 8928 raft_consensus.cc:388] T 00000000000000000000000000000000 P aeed5c2902d6455c9436c2ecab8ae4f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:04:10.543541 8928 raft_consensus.cc:482] T 00000000000000000000000000000000 P aeed5c2902d6455c9436c2ecab8ae4f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:04:10.543695 8928 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aeed5c2902d6455c9436c2ecab8ae4f9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:04:10.546162 8928 raft_consensus.cc:504] T 00000000000000000000000000000000 P aeed5c2902d6455c9436c2ecab8ae4f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeed5c2902d6455c9436c2ecab8ae4f9" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 34909 } }
I20260808 04:04:10.546612 8928 leader_election.cc:302] T 00000000000000000000000000000000 P aeed5c2902d6455c9436c2ecab8ae4f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aeed5c2902d6455c9436c2ecab8ae4f9; no voters:
I20260808 04:04:10.547127 8928 leader_election.cc:288] T 00000000000000000000000000000000 P aeed5c2902d6455c9436c2ecab8ae4f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:04:10.547235 8935 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aeed5c2902d6455c9436c2ecab8ae4f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:04:10.547887 8935 raft_consensus.cc:686] T 00000000000000000000000000000000 P aeed5c2902d6455c9436c2ecab8ae4f9 [term 1 LEADER]: Becoming Leader. State: Replica: aeed5c2902d6455c9436c2ecab8ae4f9, State: Running, Role: LEADER
I20260808 04:04:10.548393 8935 consensus_queue.cc:237] T 00000000000000000000000000000000 P aeed5c2902d6455c9436c2ecab8ae4f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeed5c2902d6455c9436c2ecab8ae4f9" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 34909 } }
I20260808 04:04:10.549038 8928 sys_catalog.cc:564] T 00000000000000000000000000000000 P aeed5c2902d6455c9436c2ecab8ae4f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:04:10.551153 8937 sys_catalog.cc:455] T 00000000000000000000000000000000 P aeed5c2902d6455c9436c2ecab8ae4f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aeed5c2902d6455c9436c2ecab8ae4f9. Latest consensus state: current_term: 1 leader_uuid: "aeed5c2902d6455c9436c2ecab8ae4f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeed5c2902d6455c9436c2ecab8ae4f9" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 34909 } } }
I20260808 04:04:10.551546 8937 sys_catalog.cc:458] T 00000000000000000000000000000000 P aeed5c2902d6455c9436c2ecab8ae4f9 [sys.catalog]: This master's current role is: LEADER
I20260808 04:04:10.552038 8936 sys_catalog.cc:455] T 00000000000000000000000000000000 P aeed5c2902d6455c9436c2ecab8ae4f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aeed5c2902d6455c9436c2ecab8ae4f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeed5c2902d6455c9436c2ecab8ae4f9" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 34909 } } }
I20260808 04:04:10.552378 8936 sys_catalog.cc:458] T 00000000000000000000000000000000 P aeed5c2902d6455c9436c2ecab8ae4f9 [sys.catalog]: This master's current role is: LEADER
I20260808 04:04:10.556586 8944 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:04:10.561012 8944 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:04:10.568693 8944 catalog_manager.cc:1347] Generated new cluster ID: eaae7245ff934ac785fc530ce9a9d104
I20260808 04:04:10.568804 8944 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:04:10.578759 8944 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:04:10.580544 8944 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:04:10.591967 8944 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aeed5c2902d6455c9436c2ecab8ae4f9: Generated new TSK 0
I20260808 04:04:10.592973 8944 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.83.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.1.keytab.
Entry for principal kudu/127.5.83.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.1.keytab.
I20260808 04:04:10.642436 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.1:0
--local_ip_for_outbound_sockets=127.5.83.1
--webserver_interface=127.5.83.1
--webserver_port=0
--tserver_master_addrs=127.5.83.62:34909
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.1.keytab
--principal=kudu/127.5.83.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:43511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:04:10.955192 8964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:04:10.955476 8964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:04:10.955606 8964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:04:10.962788 8964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:04:10.962888 8964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:04:10.963004 8964 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.1
I20260808 04:04:10.972417 8964 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:43511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.1.keytab
--principal=kudu/127.5.83.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.83.1
--webserver_port=0
--tserver_master_addrs=127.5.83.62:34909
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.1
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:04:10.973542 8964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:04:10.974732 8964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:04:10.986804 8977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:10.986850 8975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:10.986876 8974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:04:10.987646 8964 server_base.cc:1034] running on GCE node
I20260808 04:04:10.988224 8964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:04:10.989130 8964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:04:10.990315 8964 hybrid_clock.cc:648] HybridClock initialized: now 1786161850990278 us; error 30 us; skew 500 ppm
I20260808 04:04:10.990598 8964 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:04:10 dist-test-slave-1l3l krb5kdc[8810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161850, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:04:10.993652 8964 init.cc:377] Logged in from keytab as kudu/127.5.83.1@KRBTEST.COM (short username kudu)
I20260808 04:04:10.995056 8964 webserver.cc:460] Webserver started at http://127.5.83.1:43489/ using document root <none> and password file <none>
I20260808 04:04:10.995728 8964 fs_manager.cc:359] Metadata directory not provided
I20260808 04:04:10.995851 8964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:04:10.996160 8964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:04:10.999020 8964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-0/data/instance:
uuid: "9b047364c3a4448f86da93bdfe2a7844"
format_stamp: "Formatted at 2026-08-08 04:04:10 on dist-test-slave-1l3l"
server_key: "2655a1b659461129ab65430dd1bda410"
server_key_iv: "1ccb7a71b5af0a41b6dd0199768edd45"
server_key_version: "encryptionkey@0"
I20260808 04:04:10.999786 8964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-0/wal/instance:
uuid: "9b047364c3a4448f86da93bdfe2a7844"
format_stamp: "Formatted at 2026-08-08 04:04:10 on dist-test-slave-1l3l"
server_key: "2655a1b659461129ab65430dd1bda410"
server_key_iv: "1ccb7a71b5af0a41b6dd0199768edd45"
server_key_version: "encryptionkey@0"
I20260808 04:04:11.004562 8964 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260808 04:04:11.007803 8984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:11.009061 8964 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 04:04:11.009207 8964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-0/wal
uuid: "9b047364c3a4448f86da93bdfe2a7844"
format_stamp: "Formatted at 2026-08-08 04:04:10 on dist-test-slave-1l3l"
server_key: "2655a1b659461129ab65430dd1bda410"
server_key_iv: "1ccb7a71b5af0a41b6dd0199768edd45"
server_key_version: "encryptionkey@0"
I20260808 04:04:11.009409 8964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:04:11.022949 8964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:04:11.026952 8964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:04:11.027220 8964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:04:11.028231 8964 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:04:11.029973 8964 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:04:11.030068 8964 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:11.030189 8964 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:04:11.030259 8964 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:11.060750 8964 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.1:43087
I20260808 04:04:11.060788 9138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.1:43087 every 8 connection(s)
I20260808 04:04:11.062076 8964 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
I20260808 04:04:11.070303 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 8964
I20260808 04:04:11.070518 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-0/wal/instance
I20260808 04:04:11.071128 5452 external_mini_cluster.cc:1428] Setting key 0c7f8b9c736c3b03814f6927fb978e3a
Aug 08 04:04:11 dist-test-slave-1l3l krb5kdc[8810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161850, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.1@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:04:11.081269 9142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:11.064536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.1:55487 (local address 127.5.83.62:34909)
0808 04:04:11.064992 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:11.064997 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:11.065899 (+ 902us) server_negotiation.cc:407] Connection header received
0808 04:04:11.067661 (+ 1762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:11.067667 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:11.067783 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:11.067892 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:11.069333 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:11.070077 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:11.071634 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:11.072228 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:11.072339 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:11.075967 (+ 3628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:11.075972 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:11.075975 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:11.076006 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:11.078441 (+ 2435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:11.079138 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:11.079143 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:11.079145 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:11.079260 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:11.079661 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:11.079666 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:11.079669 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:11.079986 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:11.080116 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:11.080828 (+ 712us) server_negotiation.cc:299] Negotiation successful
0808 04:04:11.080994 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":233,"threads_started":1}
I20260808 04:04:11.082592 9143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:11.065045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:34909 (local address 127.5.83.1:55487)
0808 04:04:11.065747 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:11.065782 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:11.067407 (+ 1625us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:11.068121 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:11.068129 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:11.068629 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:04:11.069171 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:11.069187 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:11.070214 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:11.070217 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:11.071503 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:11.071515 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:11.072459 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:11.072467 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:11.072705 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:11.073375 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:04:11.073405 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:04:11.075752 (+ 2347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:04:11.078580 (+ 2828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:11.078588 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:11.078606 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:11.079026 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:11.079362 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:11.079367 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:11.079371 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:11.079557 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:11.080109 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:04:11.080116 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:04:11.080447 (+ 331us) client_negotiation.cc:769] Sending connection context
0808 04:04:11.080740 (+ 293us) client_negotiation.cc:240] Negotiation successful
0808 04:04:11.081271 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":282,"threads_started":1}
I20260808 04:04:11.084066 9140 heartbeater.cc:347] Connected to a master server at 127.5.83.62:34909
I20260808 04:04:11.084427 9140 heartbeater.cc:464] Registering TS with master...
I20260808 04:04:11.085237 9140 heartbeater.cc:511] Master 127.5.83.62:34909 requested a full tablet report, sending...
I20260808 04:04:11.087330 8870 ts_manager.cc:194] Registered new tserver with Master: 9b047364c3a4448f86da93bdfe2a7844 (127.5.83.1:43087)
I20260808 04:04:11.088795 8870 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.1@KRBTEST.COM'} at 127.5.83.1:55487
WARNING: no policy specified for kudu/127.5.83.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.2.keytab.
Entry for principal kudu/127.5.83.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.2.keytab.
I20260808 04:04:11.116735 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.2:0
--local_ip_for_outbound_sockets=127.5.83.2
--webserver_interface=127.5.83.2
--webserver_port=0
--tserver_master_addrs=127.5.83.62:34909
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.2.keytab
--principal=kudu/127.5.83.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:43511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:04:11.433313 9146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:04:11.433578 9146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:04:11.433668 9146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:04:11.440696 9146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:04:11.440794 9146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:04:11.440910 9146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.2
I20260808 04:04:11.450215 9146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:43511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.2.keytab
--principal=kudu/127.5.83.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.83.2
--webserver_port=0
--tserver_master_addrs=127.5.83.62:34909
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.2
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:04:11.451346 9146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:04:11.452597 9146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:04:11.464957 9153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:11.464964 9156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:11.465405 9160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:04:11.466003 9146 server_base.cc:1034] running on GCE node
I20260808 04:04:11.466526 9146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:04:11.467293 9146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:04:11.468472 9146 hybrid_clock.cc:648] HybridClock initialized: now 1786161851468423 us; error 50 us; skew 500 ppm
I20260808 04:04:11.468745 9146 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:04:11 dist-test-slave-1l3l krb5kdc[8810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161851, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:04:11.471750 9146 init.cc:377] Logged in from keytab as kudu/127.5.83.2@KRBTEST.COM (short username kudu)
I20260808 04:04:11.473070 9146 webserver.cc:460] Webserver started at http://127.5.83.2:41017/ using document root <none> and password file <none>
I20260808 04:04:11.473714 9146 fs_manager.cc:359] Metadata directory not provided
I20260808 04:04:11.473821 9146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:04:11.474114 9146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:04:11.476930 9146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-1/data/instance:
uuid: "15904ab6baf644409871a632e082e1a9"
format_stamp: "Formatted at 2026-08-08 04:04:11 on dist-test-slave-1l3l"
server_key: "b54cf709bdd34c0524a5242d331be030"
server_key_iv: "a58ac8fd376121536e6d59787ad76c70"
server_key_version: "encryptionkey@0"
I20260808 04:04:11.477666 9146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-1/wal/instance:
uuid: "15904ab6baf644409871a632e082e1a9"
format_stamp: "Formatted at 2026-08-08 04:04:11 on dist-test-slave-1l3l"
server_key: "b54cf709bdd34c0524a5242d331be030"
server_key_iv: "a58ac8fd376121536e6d59787ad76c70"
server_key_version: "encryptionkey@0"
I20260808 04:04:11.482576 9146 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260808 04:04:11.485926 9168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:11.487111 9146 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260808 04:04:11.487253 9146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-1/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-1/wal
uuid: "15904ab6baf644409871a632e082e1a9"
format_stamp: "Formatted at 2026-08-08 04:04:11 on dist-test-slave-1l3l"
server_key: "b54cf709bdd34c0524a5242d331be030"
server_key_iv: "a58ac8fd376121536e6d59787ad76c70"
server_key_version: "encryptionkey@0"
I20260808 04:04:11.487450 9146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:04:11.525313 9146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:04:11.530200 9146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:04:11.530515 9146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:04:11.531477 9146 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:04:11.532733 9146 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:04:11.532806 9146 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:11.532898 9146 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:04:11.532948 9146 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:11.562759 9146 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.2:45221
I20260808 04:04:11.562824 9330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.2:45221 every 8 connection(s)
I20260808 04:04:11.564065 9146 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
I20260808 04:04:11.564936 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 9146
I20260808 04:04:11.565127 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-1/wal/instance
I20260808 04:04:11.565564 5452 external_mini_cluster.cc:1428] Setting key 9f66dd2397f9662f0e8f0e071931ca1a
Aug 08 04:04:11 dist-test-slave-1l3l krb5kdc[8810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161851, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.2@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:04:11.582791 9142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:11.566673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.2:57807 (local address 127.5.83.62:34909)
0808 04:04:11.566829 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:11.566833 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:11.568143 (+ 1310us) server_negotiation.cc:407] Connection header received
0808 04:04:11.569884 (+ 1741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:11.569889 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:11.569969 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:11.570055 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:11.571537 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:11.572282 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:11.573754 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:11.574159 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:11.574269 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:11.577516 (+ 3247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:11.577521 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:11.577524 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:11.577555 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:11.579977 (+ 2422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:11.580618 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:11.580622 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:11.580624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:11.580758 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:11.581170 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:11.581175 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:11.581177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:11.581444 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:11.581577 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:11.582234 (+ 657us) server_negotiation.cc:299] Negotiation successful
0808 04:04:11.582474 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
WARNING: no policy specified for kudu/127.5.83.3@KRBTEST.COM; defaulting to no policy
I20260808 04:04:11.583755 9333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:11.567266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:34909 (local address 127.5.83.2:57807)
0808 04:04:11.568009 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:11.568047 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:11.569639 (+ 1592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:11.570287 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:11.570294 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:11.570773 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:04:11.571372 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:11.571390 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:11.572432 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:11.572435 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:11.573619 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:11.573631 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:11.574368 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:11.574376 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:11.574604 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:11.575202 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:04:11.575223 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:04:11.577315 (+ 2092us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:04:11.580098 (+ 2783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:11.580107 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:11.580122 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:11.580507 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:11.580863 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:11.580868 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:11.580872 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:11.581064 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:11.581568 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:04:11.581575 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:04:11.581868 (+ 293us) client_negotiation.cc:769] Sending connection context
0808 04:04:11.582111 (+ 243us) client_negotiation.cc:240] Negotiation successful
0808 04:04:11.582655 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":329,"threads_started":1}
I20260808 04:04:11.585038 9331 heartbeater.cc:347] Connected to a master server at 127.5.83.62:34909
I20260808 04:04:11.585330 9331 heartbeater.cc:464] Registering TS with master...
I20260808 04:04:11.585995 9331 heartbeater.cc:511] Master 127.5.83.62:34909 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.3@KRBTEST.COM" created.
I20260808 04:04:11.587322 8870 ts_manager.cc:194] Registered new tserver with Master: 15904ab6baf644409871a632e082e1a9 (127.5.83.2:45221)
I20260808 04:04:11.588142 8870 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.2@KRBTEST.COM'} at 127.5.83.2:57807
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.3.keytab.
Entry for principal kudu/127.5.83.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.3.keytab.
I20260808 04:04:11.608989 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.3:0
--local_ip_for_outbound_sockets=127.5.83.3
--webserver_interface=127.5.83.3
--webserver_port=0
--tserver_master_addrs=127.5.83.62:34909
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.3.keytab
--principal=kudu/127.5.83.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:43511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:04:11.918336 9339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:04:11.918601 9339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:04:11.918731 9339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:04:11.925669 9339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:04:11.925765 9339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:04:11.925910 9339 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.3
I20260808 04:04:11.935146 9339 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:43511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.3.keytab
--principal=kudu/127.5.83.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.83.3
--webserver_port=0
--tserver_master_addrs=127.5.83.62:34909
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.3
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:04:11.936323 9339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:04:11.937532 9339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:04:11.949352 9350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:11.949502 9348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:11.949327 9353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:04:11.950435 9339 server_base.cc:1034] running on GCE node
I20260808 04:04:11.950950 9339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:04:11.951814 9339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:04:11.952999 9339 hybrid_clock.cc:648] HybridClock initialized: now 1786161851952953 us; error 40 us; skew 500 ppm
I20260808 04:04:11.953279 9339 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:04:11 dist-test-slave-1l3l krb5kdc[8810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161851, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:04:11.956315 9339 init.cc:377] Logged in from keytab as kudu/127.5.83.3@KRBTEST.COM (short username kudu)
I20260808 04:04:11.957597 9339 webserver.cc:460] Webserver started at http://127.5.83.3:38095/ using document root <none> and password file <none>
I20260808 04:04:11.958189 9339 fs_manager.cc:359] Metadata directory not provided
I20260808 04:04:11.958283 9339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:04:11.958532 9339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:04:11.961269 9339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-2/data/instance:
uuid: "3942f67973e04d4ba32df47254927b15"
format_stamp: "Formatted at 2026-08-08 04:04:11 on dist-test-slave-1l3l"
server_key: "ebd9cdba00d232288b84e6d3156d563e"
server_key_iv: "84a58350066de1d7a3ac91501b83cf8c"
server_key_version: "encryptionkey@0"
I20260808 04:04:11.961978 9339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-2/wal/instance:
uuid: "3942f67973e04d4ba32df47254927b15"
format_stamp: "Formatted at 2026-08-08 04:04:11 on dist-test-slave-1l3l"
server_key: "ebd9cdba00d232288b84e6d3156d563e"
server_key_iv: "84a58350066de1d7a3ac91501b83cf8c"
server_key_version: "encryptionkey@0"
I20260808 04:04:11.966787 9339 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 04:04:11.969949 9365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:11.971161 9339 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260808 04:04:11.971313 9339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-2/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-2/wal
uuid: "3942f67973e04d4ba32df47254927b15"
format_stamp: "Formatted at 2026-08-08 04:04:11 on dist-test-slave-1l3l"
server_key: "ebd9cdba00d232288b84e6d3156d563e"
server_key_iv: "84a58350066de1d7a3ac91501b83cf8c"
server_key_version: "encryptionkey@0"
I20260808 04:04:11.971496 9339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:04:11.985052 9339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:04:11.988586 9339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:04:11.988829 9339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:04:11.989686 9339 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:04:11.990908 9339 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:04:11.990983 9339 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:11.991092 9339 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:04:11.991145 9339 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:12.020974 9339 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.3:39703
I20260808 04:04:12.021029 9520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.3:39703 every 8 connection(s)
I20260808 04:04:12.022188 9339 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
I20260808 04:04:12.025246 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 9339
I20260808 04:04:12.025444 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-2/wal/instance
I20260808 04:04:12.025853 5452 external_mini_cluster.cc:1428] Setting key c1f3e7902af81802a1aeccf93f477c14
Aug 08 04:04:12 dist-test-slave-1l3l krb5kdc[8810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161851, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.3@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:04:12.040763 9142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:12.024786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.3:35295 (local address 127.5.83.62:34909)
0808 04:04:12.025008 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:12.025015 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:12.026238 (+ 1223us) server_negotiation.cc:407] Connection header received
0808 04:04:12.028083 (+ 1845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:12.028089 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:12.028170 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:12.028259 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:12.029687 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:12.030439 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:12.031955 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:12.032354 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:12.032470 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:12.035589 (+ 3119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:12.035593 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:12.035596 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:12.035626 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:12.037975 (+ 2349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:12.038608 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:12.038614 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:12.038618 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:12.038735 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:12.039135 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:12.039140 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:12.039144 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:12.039424 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:12.039551 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:12.040222 (+ 671us) server_negotiation.cc:299] Negotiation successful
0808 04:04:12.040445 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260808 04:04:12.041594 9523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:12.025251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:34909 (local address 127.5.83.3:35295)
0808 04:04:12.026066 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:12.026112 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:12.027827 (+ 1715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:12.028505 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:12.028513 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:12.028999 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:04:12.029527 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:12.029543 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:12.030587 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:12.030591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:12.031824 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:12.031836 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:12.032559 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:12.032566 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:12.032798 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:12.033358 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:04:12.033378 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:04:12.035394 (+ 2016us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:04:12.038108 (+ 2714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:12.038117 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:12.038131 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:12.038499 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:12.038834 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:12.038839 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:12.038843 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:12.039027 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:12.039545 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:04:12.039551 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:04:12.039868 (+ 317us) client_negotiation.cc:769] Sending connection context
0808 04:04:12.040099 (+ 231us) client_negotiation.cc:240] Negotiation successful
0808 04:04:12.040552 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":336,"threads_started":1}
I20260808 04:04:12.042678 9521 heartbeater.cc:347] Connected to a master server at 127.5.83.62:34909
I20260808 04:04:12.042955 9521 heartbeater.cc:464] Registering TS with master...
I20260808 04:04:12.043634 9521 heartbeater.cc:511] Master 127.5.83.62:34909 requested a full tablet report, sending...
I20260808 04:04:12.044932 8870 ts_manager.cc:194] Registered new tserver with Master: 3942f67973e04d4ba32df47254927b15 (127.5.83.3:39703)
I20260808 04:04:12.045684 8870 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.3@KRBTEST.COM'} at 127.5.83.3:35295
I20260808 04:04:12.052196 5452 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260808 04:04:12.083658 9142 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:04:12.081271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39192 (local address 127.5.83.62:34909)
0808 04:04:12.081481 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:12.081488 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:12.081893 (+ 405us) server_negotiation.cc:407] Connection header received
0808 04:04:12.082087 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:12.082092 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:12.082156 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:12.082243 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:12.083469 (+ 1226us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39192: BlockingRecv error: recv got EOF from 127.0.0.1:39192 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 04:04:12.086763 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 8964
W20260808 04:04:12.089336 5452 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 8964: Not authorized: Client connection negotiation failed: client connection to 127.5.83.1:43087: server requires authentication, but client does not have Kerberos credentials available
I20260808 04:04:12.089489 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 8964
I20260808 04:04:12.113775 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 9146
W20260808 04:04:12.116338 5452 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 9146: Not authorized: Client connection negotiation failed: client connection to 127.5.83.2:45221: server requires authentication, but client does not have Kerberos credentials available
I20260808 04:04:12.116518 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 9146
I20260808 04:04:12.136299 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 9339
W20260808 04:04:12.139104 5452 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 9339: Not authorized: Client connection negotiation failed: client connection to 127.5.83.3:39703: server requires authentication, but client does not have Kerberos credentials available
I20260808 04:04:12.139252 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 9339
I20260808 04:04:12.157835 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 8826
I20260808 04:04:12.159009 8919 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:04:12.260164 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 8826
2026-08-08T04:04:12Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3504 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786161826605655-5452-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:04:12 dist-test-slave-1l3l krb5kdc[9544](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:04:12 dist-test-slave-1l3l krb5kdc[9544](info): set up 2 sockets
Aug 08 04:04:12 dist-test-slave-1l3l krb5kdc[9544](info): commencing operation
krb5kdc: starting...
W20260808 04:04:13.430104 5452 mini_kdc.cc:121] Time spent starting KDC: real 1.124s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786161826605655-5452-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786161826605655-5452-0/krb5kdc/test-user.keytab.
Aug 08 04:04:13 dist-test-slave-1l3l krb5kdc[9544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:04:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:04:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.83.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab.
Entry for principal kudu/127.5.83.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab.
I20260808 04:04:13.610534 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786161826605655-5452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786161826605655-5452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786161826605655-5452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:38179
--webserver_interface=127.5.83.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab
--principal=kudu/127.5.83.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:43825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.62:38179
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:04:13.917618 9560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:04:13.917872 9560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:04:13.917966 9560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:04:13.924836 9560 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:04:13.924928 9560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:04:13.924979 9560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:04:13.925037 9560 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:04:13.925083 9560 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:04:13.934206 9560 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:43825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786161826605655-5452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786161826605655-5452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.62:38179
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab
--principal=kudu/127.5.83.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:38179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786161826605655-5452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:04:13.935334 9560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:04:13.936558 9560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:04:13.947185 9566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:13.947289 9571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:13.947240 9567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:04:13.948045 9560 server_base.cc:1034] running on GCE node
I20260808 04:04:13.948880 9560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:04:13.950102 9560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:04:13.951345 9560 hybrid_clock.cc:648] HybridClock initialized: now 1786161853951269 us; error 34 us; skew 500 ppm
I20260808 04:04:13.951670 9560 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:04:13 dist-test-slave-1l3l krb5kdc[9544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161853, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:04:13.961887 9560 init.cc:377] Logged in from keytab as kudu/127.5.83.62@KRBTEST.COM (short username kudu)
I20260808 04:04:13.963223 9560 webserver.cc:460] Webserver started at http://127.5.83.62:43173/ using document root <none> and password file <none>
I20260808 04:04:13.963915 9560 fs_manager.cc:359] Metadata directory not provided
I20260808 04:04:13.964041 9560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:04:13.964319 9560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:04:13.967026 9560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786161826605655-5452-0/minicluster-data/master-0/data/instance:
uuid: "9c32aae7180f46fb865b52856f173ed5"
format_stamp: "Formatted at 2026-08-08 04:04:13 on dist-test-slave-1l3l"
server_key: "6dd92bb97d9a69f6f476f0d2f4f07746"
server_key_iv: "803143c13fda1dc30292dec0175c8b8e"
server_key_version: "encryptionkey@0"
I20260808 04:04:13.967792 9560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786161826605655-5452-0/minicluster-data/master-0/wal/instance:
uuid: "9c32aae7180f46fb865b52856f173ed5"
format_stamp: "Formatted at 2026-08-08 04:04:13 on dist-test-slave-1l3l"
server_key: "6dd92bb97d9a69f6f476f0d2f4f07746"
server_key_iv: "803143c13fda1dc30292dec0175c8b8e"
server_key_version: "encryptionkey@0"
I20260808 04:04:13.972360 9560 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260808 04:04:13.975433 9582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:13.976612 9560 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 04:04:13.976761 9560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786161826605655-5452-0/minicluster-data/master-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786161826605655-5452-0/minicluster-data/master-0/wal
uuid: "9c32aae7180f46fb865b52856f173ed5"
format_stamp: "Formatted at 2026-08-08 04:04:13 on dist-test-slave-1l3l"
server_key: "6dd92bb97d9a69f6f476f0d2f4f07746"
server_key_iv: "803143c13fda1dc30292dec0175c8b8e"
server_key_version: "encryptionkey@0"
I20260808 04:04:13.976919 9560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786161826605655-5452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786161826605655-5452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786161826605655-5452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:04:13.989406 9560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:04:13.993137 9560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:04:13.993371 9560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:04:14.009835 9560 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.62:38179
I20260808 04:04:14.009874 9652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.62:38179 every 8 connection(s)
I20260808 04:04:14.011080 9560 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
I20260808 04:04:14.014588 9653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:04:14.015806 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 9560
I20260808 04:04:14.015988 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786161826605655-5452-0/minicluster-data/master-0/wal/instance
I20260808 04:04:14.016629 5452 external_mini_cluster.cc:1428] Setting key 47f3019357b043dcde5cdaf8deda5d6c
I20260808 04:04:14.028076 9653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c32aae7180f46fb865b52856f173ed5: Bootstrap starting.
Aug 08 04:04:14 dist-test-slave-1l3l krb5kdc[9544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:04:14.031116 9653 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c32aae7180f46fb865b52856f173ed5: Neither blocks nor log segments found. Creating new log.
I20260808 04:04:14.032265 9653 log.cc:824] T 00000000000000000000000000000000 P 9c32aae7180f46fb865b52856f173ed5: Log is configured to *not* fsync() on all Append() calls
I20260808 04:04:14.034875 9653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c32aae7180f46fb865b52856f173ed5: No bootstrap required, opened a new log
I20260808 04:04:14.038743 9660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:14.018318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42068 (local address 127.5.83.62:38179)
0808 04:04:14.019221 (+ 903us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:14.019236 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:14.019288 (+ 52us) server_negotiation.cc:407] Connection header received
0808 04:04:14.020910 (+ 1622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:14.020950 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:14.021424 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:14.021858 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:14.023106 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:14.025127 (+ 2021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:14.026307 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:14.027297 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:14.027467 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:14.031865 (+ 4398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:14.031873 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:14.031886 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:14.031920 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:14.034998 (+ 3078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:14.035629 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:14.035636 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:14.035641 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:14.035841 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:14.036186 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:14.036191 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:14.036195 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:14.036722 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:14.037002 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:14.037448 (+ 446us) server_negotiation.cc:299] Negotiation successful
0808 04:04:14.037817 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":652,"thread_start_us":390,"threads_started":1}
I20260808 04:04:14.042454 5452 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:04:14.043008 9653 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c32aae7180f46fb865b52856f173ed5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c32aae7180f46fb865b52856f173ed5" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 38179 } }
I20260808 04:04:14.043308 9653 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c32aae7180f46fb865b52856f173ed5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:04:14.043401 9653 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c32aae7180f46fb865b52856f173ed5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c32aae7180f46fb865b52856f173ed5, State: Initialized, Role: FOLLOWER
I20260808 04:04:14.043934 9653 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c32aae7180f46fb865b52856f173ed5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c32aae7180f46fb865b52856f173ed5" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 38179 } }
I20260808 04:04:14.044142 9653 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c32aae7180f46fb865b52856f173ed5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:04:14.044261 9653 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c32aae7180f46fb865b52856f173ed5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:04:14.044421 9653 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c32aae7180f46fb865b52856f173ed5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:04:14.046384 9653 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c32aae7180f46fb865b52856f173ed5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c32aae7180f46fb865b52856f173ed5" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 38179 } }
I20260808 04:04:14.046836 9653 leader_election.cc:302] T 00000000000000000000000000000000 P 9c32aae7180f46fb865b52856f173ed5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c32aae7180f46fb865b52856f173ed5; no voters:
I20260808 04:04:14.047425 9653 leader_election.cc:288] T 00000000000000000000000000000000 P 9c32aae7180f46fb865b52856f173ed5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:04:14.047506 9663 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c32aae7180f46fb865b52856f173ed5 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:04:14.049507 9653 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c32aae7180f46fb865b52856f173ed5 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:04:14.050112 9663 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c32aae7180f46fb865b52856f173ed5 [term 1 LEADER]: Becoming Leader. State: Replica: 9c32aae7180f46fb865b52856f173ed5, State: Running, Role: LEADER
I20260808 04:04:14.050624 9663 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c32aae7180f46fb865b52856f173ed5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c32aae7180f46fb865b52856f173ed5" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 38179 } }
I20260808 04:04:14.054564 9666 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c32aae7180f46fb865b52856f173ed5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c32aae7180f46fb865b52856f173ed5. Latest consensus state: current_term: 1 leader_uuid: "9c32aae7180f46fb865b52856f173ed5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c32aae7180f46fb865b52856f173ed5" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 38179 } } }
I20260808 04:04:14.054913 9666 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c32aae7180f46fb865b52856f173ed5 [sys.catalog]: This master's current role is: LEADER
I20260808 04:04:14.055455 9664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c32aae7180f46fb865b52856f173ed5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9c32aae7180f46fb865b52856f173ed5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c32aae7180f46fb865b52856f173ed5" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 38179 } } }
I20260808 04:04:14.055749 9664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c32aae7180f46fb865b52856f173ed5 [sys.catalog]: This master's current role is: LEADER
I20260808 04:04:14.062083 9685 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:04:14.067029 9685 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:04:14.075838 9685 catalog_manager.cc:1347] Generated new cluster ID: 673de918578747f6b3aea3f073c071da
I20260808 04:04:14.075947 9685 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:04:14.087352 9685 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:04:14.088853 9685 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:04:14.097088 9660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:14.085667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42076 (local address 127.5.83.62:38179)
0808 04:04:14.085823 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:14.085829 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:14.088312 (+ 2483us) server_negotiation.cc:407] Connection header received
0808 04:04:14.088497 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:14.088503 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:14.088892 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:14.089016 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:14.090419 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:14.091487 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:14.093353 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:14.094027 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:14.094133 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:14.095325 (+ 1192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:14.095333 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:14.095338 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:04:14.096276 (+ 938us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:14.096346 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:04:14.096378 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:14.096479 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:14.096640 (+ 161us) server_negotiation.cc:299] Negotiation successful
0808 04:04:14.096731 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":51}
I20260808 04:04:14.105202 9685 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9c32aae7180f46fb865b52856f173ed5: Generated new TSK 0
I20260808 04:04:14.106192 9685 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:04:14.117362 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 9560
I20260808 04:04:14.118510 9647 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:04:14.206962 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 9560
2026-08-08T04:04:14Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1940 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:04:14 dist-test-slave-1l3l krb5kdc[9706](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:04:14 dist-test-slave-1l3l krb5kdc[9706](info): set up 2 sockets
Aug 08 04:04:14 dist-test-slave-1l3l krb5kdc[9706](info): commencing operation
krb5kdc: starting...
W20260808 04:04:15.364061 5452 mini_kdc.cc:121] Time spent starting KDC: real 1.117s user 0.002s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/krb5kdc/test-user.keytab.
Aug 08 04:04:15 dist-test-slave-1l3l krb5kdc[9706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:04:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:04:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.83.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab.
Entry for principal kudu/127.5.83.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab.
I20260808 04:04:15.547006 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:46153
--webserver_interface=127.5.83.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab
--principal=kudu/127.5.83.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:33569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.62:46153
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:04:15.859323 9724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:04:15.859596 9724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:04:15.859737 9724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:04:15.866626 9724 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:04:15.866724 9724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:04:15.866780 9724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:04:15.866858 9724 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:04:15.866909 9724 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:04:15.876277 9724 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:33569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.62:46153
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab
--principal=kudu/127.5.83.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:46153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:04:15.877540 9724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:04:15.878778 9724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:04:15.889853 9730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:15.889887 9731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:15.890096 9733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:04:15.890625 9724 server_base.cc:1034] running on GCE node
I20260808 04:04:15.891395 9724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:04:15.892741 9724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:04:15.893944 9724 hybrid_clock.cc:648] HybridClock initialized: now 1786161855893883 us; error 45 us; skew 500 ppm
I20260808 04:04:15.894294 9724 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:04:15 dist-test-slave-1l3l krb5kdc[9706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161855, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:04:15.897277 9724 init.cc:377] Logged in from keytab as kudu/127.5.83.62@KRBTEST.COM (short username kudu)
I20260808 04:04:15.898706 9724 webserver.cc:460] Webserver started at http://127.5.83.62:45495/ using document root <none> and password file <none>
I20260808 04:04:15.899317 9724 fs_manager.cc:359] Metadata directory not provided
I20260808 04:04:15.899444 9724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:04:15.899775 9724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:04:15.902602 9724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/minicluster-data/master-0/data/instance:
uuid: "a5460931e2cd4b1ebe8c10107b781b08"
format_stamp: "Formatted at 2026-08-08 04:04:15 on dist-test-slave-1l3l"
server_key: "fe7bcacf6a5d92faa1bd4b83d10c37c0"
server_key_iv: "63b4de7c4961923f607d4fde1a4c6ccc"
server_key_version: "encryptionkey@0"
I20260808 04:04:15.903313 9724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/minicluster-data/master-0/wal/instance:
uuid: "a5460931e2cd4b1ebe8c10107b781b08"
format_stamp: "Formatted at 2026-08-08 04:04:15 on dist-test-slave-1l3l"
server_key: "fe7bcacf6a5d92faa1bd4b83d10c37c0"
server_key_iv: "63b4de7c4961923f607d4fde1a4c6ccc"
server_key_version: "encryptionkey@0"
I20260808 04:04:15.908111 9724 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260808 04:04:15.911294 9741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:15.912604 9724 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:04:15.912762 9724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/minicluster-data/master-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/minicluster-data/master-0/wal
uuid: "a5460931e2cd4b1ebe8c10107b781b08"
format_stamp: "Formatted at 2026-08-08 04:04:15 on dist-test-slave-1l3l"
server_key: "fe7bcacf6a5d92faa1bd4b83d10c37c0"
server_key_iv: "63b4de7c4961923f607d4fde1a4c6ccc"
server_key_version: "encryptionkey@0"
I20260808 04:04:15.912933 9724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:04:15.954037 9724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:04:15.957850 9724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:04:15.958113 9724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:04:15.975103 9724 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.62:46153
I20260808 04:04:15.975136 9818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.62:46153 every 8 connection(s)
I20260808 04:04:15.976384 9724 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
I20260808 04:04:15.979835 9819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:04:15.982975 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 9724
I20260808 04:04:15.983187 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/minicluster-data/master-0/wal/instance
I20260808 04:04:15.984041 5452 external_mini_cluster.cc:1428] Setting key d451e0e54077b8d08b9761a9fb261dea
I20260808 04:04:15.992182 9819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5460931e2cd4b1ebe8c10107b781b08: Bootstrap starting.
I20260808 04:04:15.995646 9819 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a5460931e2cd4b1ebe8c10107b781b08: Neither blocks nor log segments found. Creating new log.
I20260808 04:04:15.996902 9819 log.cc:824] T 00000000000000000000000000000000 P a5460931e2cd4b1ebe8c10107b781b08: Log is configured to *not* fsync() on all Append() calls
Aug 08 04:04:15 dist-test-slave-1l3l krb5kdc[9706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:04:15.999974 9819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5460931e2cd4b1ebe8c10107b781b08: No bootstrap required, opened a new log
I20260808 04:04:16.006377 9824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:15.986027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60378 (local address 127.5.83.62:46153)
0808 04:04:15.986736 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:15.986746 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:15.986788 (+ 42us) server_negotiation.cc:407] Connection header received
0808 04:04:15.988403 (+ 1615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:15.988426 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:15.988782 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:15.989093 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:15.990799 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:15.992119 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:15.993799 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:15.994449 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:15.994571 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:15.999456 (+ 4885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:15.999464 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:15.999476 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:15.999509 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:16.002593 (+ 3084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:16.003352 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:16.003361 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:16.003367 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:16.003531 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:16.003941 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:16.003945 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:16.003947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:16.004460 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:16.004714 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:16.005089 (+ 375us) server_negotiation.cc:299] Negotiation successful
0808 04:04:16.005390 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":335,"threads_started":1}
I20260808 04:04:16.008210 9819 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5460931e2cd4b1ebe8c10107b781b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5460931e2cd4b1ebe8c10107b781b08" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 46153 } }
I20260808 04:04:16.008553 9819 raft_consensus.cc:374] T 00000000000000000000000000000000 P a5460931e2cd4b1ebe8c10107b781b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:04:16.008654 9819 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5460931e2cd4b1ebe8c10107b781b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5460931e2cd4b1ebe8c10107b781b08, State: Initialized, Role: FOLLOWER
I20260808 04:04:16.009217 9819 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5460931e2cd4b1ebe8c10107b781b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5460931e2cd4b1ebe8c10107b781b08" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 46153 } }
I20260808 04:04:16.009433 9819 raft_consensus.cc:388] T 00000000000000000000000000000000 P a5460931e2cd4b1ebe8c10107b781b08 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:04:16.009553 9819 raft_consensus.cc:482] T 00000000000000000000000000000000 P a5460931e2cd4b1ebe8c10107b781b08 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:04:16.009708 9819 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5460931e2cd4b1ebe8c10107b781b08 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:04:16.011863 9819 raft_consensus.cc:504] T 00000000000000000000000000000000 P a5460931e2cd4b1ebe8c10107b781b08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5460931e2cd4b1ebe8c10107b781b08" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 46153 } }
I20260808 04:04:16.012305 9819 leader_election.cc:302] T 00000000000000000000000000000000 P a5460931e2cd4b1ebe8c10107b781b08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a5460931e2cd4b1ebe8c10107b781b08; no voters:
I20260808 04:04:16.012876 9819 leader_election.cc:288] T 00000000000000000000000000000000 P a5460931e2cd4b1ebe8c10107b781b08 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:04:16.012976 9828 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a5460931e2cd4b1ebe8c10107b781b08 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:04:16.014240 9828 raft_consensus.cc:686] T 00000000000000000000000000000000 P a5460931e2cd4b1ebe8c10107b781b08 [term 1 LEADER]: Becoming Leader. State: Replica: a5460931e2cd4b1ebe8c10107b781b08, State: Running, Role: LEADER
I20260808 04:04:16.014822 9828 consensus_queue.cc:237] T 00000000000000000000000000000000 P a5460931e2cd4b1ebe8c10107b781b08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5460931e2cd4b1ebe8c10107b781b08" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 46153 } }
I20260808 04:04:16.014966 9819 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5460931e2cd4b1ebe8c10107b781b08 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:04:16.017658 9830 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5460931e2cd4b1ebe8c10107b781b08 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5460931e2cd4b1ebe8c10107b781b08. Latest consensus state: current_term: 1 leader_uuid: "a5460931e2cd4b1ebe8c10107b781b08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5460931e2cd4b1ebe8c10107b781b08" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 46153 } } }
I20260808 04:04:16.017722 9829 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5460931e2cd4b1ebe8c10107b781b08 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a5460931e2cd4b1ebe8c10107b781b08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5460931e2cd4b1ebe8c10107b781b08" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 46153 } } }
I20260808 04:04:16.018095 9830 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5460931e2cd4b1ebe8c10107b781b08 [sys.catalog]: This master's current role is: LEADER
I20260808 04:04:16.018158 9829 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5460931e2cd4b1ebe8c10107b781b08 [sys.catalog]: This master's current role is: LEADER
I20260808 04:04:16.019040 9843 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:04:16.023903 9843 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:04:16.032534 9843 catalog_manager.cc:1347] Generated new cluster ID: 0197bac3dce046108d2bc43d0ec7b575
I20260808 04:04:16.032691 9843 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:04:16.064642 9843 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:04:16.066560 9843 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:04:16.075872 9843 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a5460931e2cd4b1ebe8c10107b781b08: Generated new TSK 0
I20260808 04:04:16.076845 9843 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.83.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.1.keytab.
Entry for principal kudu/127.5.83.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.1.keytab.
I20260808 04:04:16.122871 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.1:0
--local_ip_for_outbound_sockets=127.5.83.1
--webserver_interface=127.5.83.1
--webserver_port=0
--tserver_master_addrs=127.5.83.62:46153
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.1.keytab
--principal=kudu/127.5.83.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:33569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:04:16.458351 9858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:04:16.458635 9858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:04:16.458777 9858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:04:16.466070 9858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:04:16.466187 9858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:04:16.466351 9858 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.1
I20260808 04:04:16.475685 9858 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:33569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.1.keytab
--principal=kudu/127.5.83.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.83.1
--webserver_port=0
--tserver_master_addrs=127.5.83.62:46153
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.1
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:04:16.476907 9858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:04:16.478132 9858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:04:16.490551 9863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:16.490595 9864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:16.490576 9866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:04:16.491664 9858 server_base.cc:1034] running on GCE node
I20260808 04:04:16.492240 9858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:04:16.493094 9858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:04:16.494278 9858 hybrid_clock.cc:648] HybridClock initialized: now 1786161856494223 us; error 52 us; skew 500 ppm
I20260808 04:04:16.494567 9858 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:04:16 dist-test-slave-1l3l krb5kdc[9706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161856, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:04:16.497553 9858 init.cc:377] Logged in from keytab as kudu/127.5.83.1@KRBTEST.COM (short username kudu)
I20260808 04:04:16.498839 9858 webserver.cc:460] Webserver started at http://127.5.83.1:38649/ using document root <none> and password file <none>
I20260808 04:04:16.499455 9858 fs_manager.cc:359] Metadata directory not provided
I20260808 04:04:16.499565 9858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:04:16.499900 9858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:04:16.502596 9858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-0/data/instance:
uuid: "b4f9650825ba4da8a3fd66e04b293f1c"
format_stamp: "Formatted at 2026-08-08 04:04:16 on dist-test-slave-1l3l"
server_key: "212ce3e4728f441efdffecf7624863ab"
server_key_iv: "7e30cbfd2c66280fc1b936c7927ae39e"
server_key_version: "encryptionkey@0"
I20260808 04:04:16.503544 9858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-0/wal/instance:
uuid: "b4f9650825ba4da8a3fd66e04b293f1c"
format_stamp: "Formatted at 2026-08-08 04:04:16 on dist-test-slave-1l3l"
server_key: "212ce3e4728f441efdffecf7624863ab"
server_key_iv: "7e30cbfd2c66280fc1b936c7927ae39e"
server_key_version: "encryptionkey@0"
I20260808 04:04:16.510057 9858 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.002s
I20260808 04:04:16.514405 9875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:16.516126 9858 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260808 04:04:16.516304 9858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-0/wal
uuid: "b4f9650825ba4da8a3fd66e04b293f1c"
format_stamp: "Formatted at 2026-08-08 04:04:16 on dist-test-slave-1l3l"
server_key: "212ce3e4728f441efdffecf7624863ab"
server_key_iv: "7e30cbfd2c66280fc1b936c7927ae39e"
server_key_version: "encryptionkey@0"
I20260808 04:04:16.516484 9858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:04:16.531726 9858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:04:16.535209 9858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:04:16.535483 9858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:04:16.536451 9858 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:04:16.537696 9858 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:04:16.537766 9858 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:16.537910 9858 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:04:16.537962 9858 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:16.567520 9858 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.1:42669
I20260808 04:04:16.567567 10021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.1:42669 every 8 connection(s)
I20260808 04:04:16.568719 9858 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
I20260808 04:04:16.572882 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 9858
I20260808 04:04:16.573063 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-0/wal/instance
I20260808 04:04:16.573690 5452 external_mini_cluster.cc:1428] Setting key 0b06c9ce58a56e34d7d5c6dd48624981
Aug 08 04:04:16 dist-test-slave-1l3l krb5kdc[9706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161856, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.1@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:04:16.587541 10025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:16.571157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.1:46425 (local address 127.5.83.62:46153)
0808 04:04:16.571600 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:16.571605 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:16.572603 (+ 998us) server_negotiation.cc:407] Connection header received
0808 04:04:16.574380 (+ 1777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:16.574387 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:16.574468 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:16.574564 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:16.576225 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:16.576938 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:16.578501 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:16.578895 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:16.579006 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:16.582328 (+ 3322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:16.582333 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:16.582336 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:16.582367 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:16.584755 (+ 2388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:16.585413 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:16.585417 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:16.585419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:16.585532 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:16.585932 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:16.585936 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:16.585938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:16.586189 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:16.586318 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:16.586967 (+ 649us) server_negotiation.cc:299] Negotiation successful
0808 04:04:16.587180 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":229,"threads_started":1}
I20260808 04:04:16.588393 10026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:16.571684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:46153 (local address 127.5.83.1:46425)
0808 04:04:16.572451 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:16.572489 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:16.574124 (+ 1635us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:16.574821 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:16.574832 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:16.575438 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:04:16.576065 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:16.576084 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:16.577093 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:16.577097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:16.578367 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:16.578381 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:16.579103 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:16.579111 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:16.579331 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:16.580029 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:04:16.580052 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:04:16.582124 (+ 2072us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:04:16.584897 (+ 2773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:16.584903 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:16.584914 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:16.585284 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:16.585632 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:16.585636 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:16.585640 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:16.585816 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:16.586308 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:04:16.586315 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:04:16.586608 (+ 293us) client_negotiation.cc:769] Sending connection context
0808 04:04:16.586828 (+ 220us) client_negotiation.cc:240] Negotiation successful
0808 04:04:16.587299 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":268,"threads_started":1}
I20260808 04:04:16.589804 10023 heartbeater.cc:347] Connected to a master server at 127.5.83.62:46153
I20260808 04:04:16.590072 10023 heartbeater.cc:464] Registering TS with master...
I20260808 04:04:16.590754 10023 heartbeater.cc:511] Master 127.5.83.62:46153 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.5.83.2@KRBTEST.COM; defaulting to no policy
I20260808 04:04:16.592960 9770 ts_manager.cc:194] Registered new tserver with Master: b4f9650825ba4da8a3fd66e04b293f1c (127.5.83.1:42669)
I20260808 04:04:16.594425 9770 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.1@KRBTEST.COM'} at 127.5.83.1:46425
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.2.keytab.
Entry for principal kudu/127.5.83.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.2.keytab.
I20260808 04:04:16.619059 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.2:0
--local_ip_for_outbound_sockets=127.5.83.2
--webserver_interface=127.5.83.2
--webserver_port=0
--tserver_master_addrs=127.5.83.62:46153
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.2.keytab
--principal=kudu/127.5.83.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:33569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:04:16.934906 10030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:04:16.935163 10030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:04:16.935282 10030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:04:16.942435 10030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:04:16.942533 10030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:04:16.942675 10030 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.2
I20260808 04:04:16.951994 10030 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:33569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.2.keytab
--principal=kudu/127.5.83.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.83.2
--webserver_port=0
--tserver_master_addrs=127.5.83.62:46153
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.2
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:04:16.953171 10030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:04:16.954404 10030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:04:16.966593 10041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:16.966708 10038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:16.966708 10039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:04:16.967645 10030 server_base.cc:1034] running on GCE node
I20260808 04:04:16.968209 10030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:04:16.969118 10030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:04:16.970316 10030 hybrid_clock.cc:648] HybridClock initialized: now 1786161856970280 us; error 29 us; skew 500 ppm
I20260808 04:04:16.970600 10030 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:04:16 dist-test-slave-1l3l krb5kdc[9706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161856, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:04:16.973546 10030 init.cc:377] Logged in from keytab as kudu/127.5.83.2@KRBTEST.COM (short username kudu)
I20260808 04:04:16.974872 10030 webserver.cc:460] Webserver started at http://127.5.83.2:34869/ using document root <none> and password file <none>
I20260808 04:04:16.975495 10030 fs_manager.cc:359] Metadata directory not provided
I20260808 04:04:16.975596 10030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:04:16.975921 10030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:04:16.978571 10030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-1/data/instance:
uuid: "ed81330b7a3c4af09bb88c48a99d8e89"
format_stamp: "Formatted at 2026-08-08 04:04:16 on dist-test-slave-1l3l"
server_key: "f2e07b122aa9375901d18779edbfb666"
server_key_iv: "938fb53bfc9b6c0296fabb09db7b6483"
server_key_version: "encryptionkey@0"
I20260808 04:04:16.979287 10030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-1/wal/instance:
uuid: "ed81330b7a3c4af09bb88c48a99d8e89"
format_stamp: "Formatted at 2026-08-08 04:04:16 on dist-test-slave-1l3l"
server_key: "f2e07b122aa9375901d18779edbfb666"
server_key_iv: "938fb53bfc9b6c0296fabb09db7b6483"
server_key_version: "encryptionkey@0"
I20260808 04:04:16.984156 10030 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260808 04:04:16.987423 10049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:16.988691 10030 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:04:16.988824 10030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-1/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-1/wal
uuid: "ed81330b7a3c4af09bb88c48a99d8e89"
format_stamp: "Formatted at 2026-08-08 04:04:16 on dist-test-slave-1l3l"
server_key: "f2e07b122aa9375901d18779edbfb666"
server_key_iv: "938fb53bfc9b6c0296fabb09db7b6483"
server_key_version: "encryptionkey@0"
I20260808 04:04:16.988999 10030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:04:17.012789 10030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:04:17.017175 10030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:04:17.017452 10030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:04:17.018393 10030 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:04:17.019788 10030 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:04:17.019860 10030 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:17.019982 10030 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:04:17.020033 10030 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:17.064426 10030 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.2:43429
I20260808 04:04:17.064478 10207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.2:43429 every 8 connection(s)
I20260808 04:04:17.065644 10030 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
I20260808 04:04:17.068138 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 10030
I20260808 04:04:17.068310 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-1/wal/instance
I20260808 04:04:17.068745 5452 external_mini_cluster.cc:1428] Setting key d8ca513800831d732bfbad53c7959c4c
Aug 08 04:04:17 dist-test-slave-1l3l krb5kdc[9706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161856, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.2@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:04:17.086294 10025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:17.068217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.2:49943 (local address 127.5.83.62:46153)
0808 04:04:17.069505 (+ 1288us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:17.069510 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:17.069694 (+ 184us) server_negotiation.cc:407] Connection header received
0808 04:04:17.071944 (+ 2250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:17.071948 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:17.072014 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:17.072095 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:17.074004 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:17.074733 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:17.076830 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:17.077218 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:17.077308 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:17.080888 (+ 3580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:17.080893 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:17.080896 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:17.080927 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:17.083340 (+ 2413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:17.084106 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:17.084110 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:17.084112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:17.084237 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:17.084652 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:17.084655 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:17.084657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:17.084926 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:17.085059 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:17.085720 (+ 661us) server_negotiation.cc:299] Negotiation successful
0808 04:04:17.085948 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1153}
I20260808 04:04:17.087278 10210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:17.068661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:46153 (local address 127.5.83.2:49943)
0808 04:04:17.069542 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:17.069589 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:17.071628 (+ 2039us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:17.072360 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:17.072371 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:17.073093 (+ 722us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:04:17.073847 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:17.073870 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:17.074887 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:17.074892 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:17.076664 (+ 1772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:17.076679 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:17.077435 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:17.077441 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:17.077703 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:17.078362 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:04:17.078389 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:04:17.080676 (+ 2287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:04:17.083470 (+ 2794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:17.083479 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:17.083495 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:17.083986 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:17.084349 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:17.084354 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:17.084358 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:17.084537 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:17.085059 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:04:17.085066 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:04:17.085364 (+ 298us) client_negotiation.cc:769] Sending connection context
0808 04:04:17.085599 (+ 235us) client_negotiation.cc:240] Negotiation successful
0808 04:04:17.086114 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":307,"threads_started":1}
I20260808 04:04:17.088716 10208 heartbeater.cc:347] Connected to a master server at 127.5.83.62:46153
I20260808 04:04:17.089004 10208 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.5.83.3@KRBTEST.COM; defaulting to no policy
I20260808 04:04:17.089720 10208 heartbeater.cc:511] Master 127.5.83.62:46153 requested a full tablet report, sending...
I20260808 04:04:17.091153 9770 ts_manager.cc:194] Registered new tserver with Master: ed81330b7a3c4af09bb88c48a99d8e89 (127.5.83.2:43429)
I20260808 04:04:17.092056 9770 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.2@KRBTEST.COM'} at 127.5.83.2:49943
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.3.keytab.
Entry for principal kudu/127.5.83.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.3.keytab.
I20260808 04:04:17.115083 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.3:0
--local_ip_for_outbound_sockets=127.5.83.3
--webserver_interface=127.5.83.3
--webserver_port=0
--tserver_master_addrs=127.5.83.62:46153
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.3.keytab
--principal=kudu/127.5.83.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:33569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:04:17.426743 10214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:04:17.426994 10214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:04:17.427112 10214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:04:17.434069 10214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:04:17.434161 10214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:04:17.434304 10214 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.3
I20260808 04:04:17.443573 10214 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:33569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.3.keytab
--principal=kudu/127.5.83.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.83.3
--webserver_port=0
--tserver_master_addrs=127.5.83.62:46153
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.3
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:04:17.444820 10214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:04:17.446064 10214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:04:17.458721 10223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:17.458804 10219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:17.458837 10228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:04:17.459625 10214 server_base.cc:1034] running on GCE node
I20260808 04:04:17.460233 10214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:04:17.461163 10214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:04:17.462399 10214 hybrid_clock.cc:648] HybridClock initialized: now 1786161857462324 us; error 47 us; skew 500 ppm
I20260808 04:04:17.462689 10214 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:04:17 dist-test-slave-1l3l krb5kdc[9706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161857, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:04:17.465873 10214 init.cc:377] Logged in from keytab as kudu/127.5.83.3@KRBTEST.COM (short username kudu)
I20260808 04:04:17.467199 10214 webserver.cc:460] Webserver started at http://127.5.83.3:38609/ using document root <none> and password file <none>
I20260808 04:04:17.467911 10214 fs_manager.cc:359] Metadata directory not provided
I20260808 04:04:17.468034 10214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:04:17.468313 10214 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:04:17.471081 10214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-2/data/instance:
uuid: "10a1b89c57b048cca692d78401241575"
format_stamp: "Formatted at 2026-08-08 04:04:17 on dist-test-slave-1l3l"
server_key: "2454ef9f776a9aa096fc47430eedb125"
server_key_iv: "18acc9315c7e6612ef1d3882eb972201"
server_key_version: "encryptionkey@0"
I20260808 04:04:17.471903 10214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-2/wal/instance:
uuid: "10a1b89c57b048cca692d78401241575"
format_stamp: "Formatted at 2026-08-08 04:04:17 on dist-test-slave-1l3l"
server_key: "2454ef9f776a9aa096fc47430eedb125"
server_key_iv: "18acc9315c7e6612ef1d3882eb972201"
server_key_version: "encryptionkey@0"
I20260808 04:04:17.476681 10214 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260808 04:04:17.479882 10238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:17.481104 10214 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 04:04:17.481274 10214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-2/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-2/wal
uuid: "10a1b89c57b048cca692d78401241575"
format_stamp: "Formatted at 2026-08-08 04:04:17 on dist-test-slave-1l3l"
server_key: "2454ef9f776a9aa096fc47430eedb125"
server_key_iv: "18acc9315c7e6612ef1d3882eb972201"
server_key_version: "encryptionkey@0"
I20260808 04:04:17.481446 10214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:04:17.500416 10214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:04:17.504240 10214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:04:17.504495 10214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:04:17.505344 10214 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:04:17.506664 10214 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:04:17.506747 10214 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:17.506868 10214 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:04:17.506930 10214 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:17.536079 10214 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.3:44357
I20260808 04:04:17.536123 10382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.3:44357 every 8 connection(s)
I20260808 04:04:17.537251 10214 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
I20260808 04:04:17.541771 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 10214
I20260808 04:04:17.541965 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161826605655-5452-0/minicluster-data/ts-2/wal/instance
I20260808 04:04:17.542501 5452 external_mini_cluster.cc:1428] Setting key 0e7ec5b55d40b08abcd66d6924c79b0f
Aug 08 04:04:17 dist-test-slave-1l3l krb5kdc[9706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161857, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.3@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:04:17.556531 10025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:17.539633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.3:55429 (local address 127.5.83.62:46153)
0808 04:04:17.539850 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:17.539854 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:17.540961 (+ 1107us) server_negotiation.cc:407] Connection header received
0808 04:04:17.543227 (+ 2266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:17.543232 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:17.543312 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:17.543399 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:17.545060 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:17.545954 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:17.547454 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:17.547906 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:17.548014 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:17.551191 (+ 3177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:17.551196 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:17.551199 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:17.551230 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:17.553656 (+ 2426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:17.554292 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:17.554298 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:17.554301 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:17.554429 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:17.554816 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:17.554821 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:17.554824 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:17.555086 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:17.555222 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:17.556004 (+ 782us) server_negotiation.cc:299] Negotiation successful
0808 04:04:17.556194 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260808 04:04:17.557570 10385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:17.540118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:46153 (local address 127.5.83.3:55429)
0808 04:04:17.540807 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:17.540851 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:17.542951 (+ 2100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:17.543642 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:17.543650 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:17.544351 (+ 701us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:04:17.544897 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:17.544916 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:17.546104 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:17.546108 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:17.547287 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:17.547300 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:17.548132 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:17.548139 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:17.548374 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:17.548973 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:04:17.549000 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:04:17.551002 (+ 2002us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:04:17.553780 (+ 2778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:17.553788 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:17.553803 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:17.554183 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:17.554531 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:17.554536 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:17.554540 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:17.554711 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:17.555213 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:04:17.555220 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:04:17.555550 (+ 330us) client_negotiation.cc:769] Sending connection context
0808 04:04:17.555883 (+ 333us) client_negotiation.cc:240] Negotiation successful
0808 04:04:17.556337 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":277,"threads_started":1}
I20260808 04:04:17.558766 10383 heartbeater.cc:347] Connected to a master server at 127.5.83.62:46153
I20260808 04:04:17.559031 10383 heartbeater.cc:464] Registering TS with master...
I20260808 04:04:17.559744 10383 heartbeater.cc:511] Master 127.5.83.62:46153 requested a full tablet report, sending...
I20260808 04:04:17.561048 9770 ts_manager.cc:194] Registered new tserver with Master: 10a1b89c57b048cca692d78401241575 (127.5.83.3:44357)
I20260808 04:04:17.561797 9770 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.3@KRBTEST.COM'} at 127.5.83.3:55429
I20260808 04:04:17.569561 5452 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 04:04:17 dist-test-slave-1l3l krb5kdc[9706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161857, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
I20260808 04:04:17.597846 10023 heartbeater.cc:503] Master 127.5.83.62:46153 was elected leader, sending a full tablet report...
Aug 08 04:04:17 dist-test-slave-1l3l krb5kdc[9706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161857, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:04:17.619011 10025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:17.605042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60394 (local address 127.5.83.62:46153)
0808 04:04:17.605244 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:17.605250 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:17.605772 (+ 522us) server_negotiation.cc:407] Connection header received
0808 04:04:17.605931 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:17.605935 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:17.605996 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:17.606082 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:17.607498 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:17.608284 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:17.609540 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:17.609939 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:17.610058 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:17.613858 (+ 3800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:17.613863 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:17.613866 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:17.613899 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:17.616735 (+ 2836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:17.617290 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:17.617296 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:17.617300 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:17.617429 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:17.617751 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:17.617754 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:17.617756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:17.618064 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:17.618238 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:17.618544 (+ 306us) server_negotiation.cc:299] Negotiation successful
0808 04:04:17.618718 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260808 04:04:17.619545 9770 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:60394
I20260808 04:04:17.621186 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 9858
Aug 08 04:04:17 dist-test-slave-1l3l krb5kdc[9706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161857, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.5.83.1@KRBTEST.COM
I20260808 04:04:17.636920 10399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:17.622501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40984 (local address 127.5.83.1:42669)
0808 04:04:17.623189 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:17.623196 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:17.623221 (+ 25us) server_negotiation.cc:407] Connection header received
0808 04:04:17.623276 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:17.623282 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:17.623493 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:17.623615 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:17.624719 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:17.625750 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:17.626913 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:17.627481 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:17.627584 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:17.631491 (+ 3907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:17.631503 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:17.631518 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:17.631552 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:17.634334 (+ 2782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:17.634975 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:17.634982 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:17.634984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:17.635124 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:17.635458 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:17.635461 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:17.635463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:17.635872 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:17.636102 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:17.636342 (+ 240us) server_negotiation.cc:299] Negotiation successful
0808 04:04:17.636541 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":354,"threads_started":1}
W20260808 04:04:17.638221 10017 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:40984
W20260808 04:04:17.639330 5452 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 9858: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 04:04:17.639523 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 9858
I20260808 04:04:17.661036 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 10030
Aug 08 04:04:17 dist-test-slave-1l3l krb5kdc[9706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161857, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.5.83.2@KRBTEST.COM
I20260808 04:04:17.676397 10400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:17.661818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58168 (local address 127.5.83.2:43429)
0808 04:04:17.662303 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:17.662310 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:17.662341 (+ 31us) server_negotiation.cc:407] Connection header received
0808 04:04:17.662401 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:17.662407 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:17.662609 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:17.662750 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:17.664052 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:17.665046 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:17.666240 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:17.666763 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:17.666889 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:17.670815 (+ 3926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:17.670827 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:17.670840 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:17.670880 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:17.673612 (+ 2732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:17.674192 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:17.674199 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:17.674201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:17.674326 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:17.674667 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:17.674672 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:17.674674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:17.675067 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:17.675410 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:17.675789 (+ 379us) server_negotiation.cc:299] Negotiation successful
0808 04:04:17.676045 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":225,"threads_started":1}
W20260808 04:04:17.677568 10200 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:58168
W20260808 04:04:17.678634 5452 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 10030: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 04:04:17.678763 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 10030
I20260808 04:04:17.697991 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 10214
Aug 08 04:04:17 dist-test-slave-1l3l krb5kdc[9706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161857, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.5.83.3@KRBTEST.COM
I20260808 04:04:17.713245 10401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:17.698827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35116 (local address 127.5.83.3:44357)
0808 04:04:17.699485 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:17.699492 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:17.699521 (+ 29us) server_negotiation.cc:407] Connection header received
0808 04:04:17.699579 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:17.699585 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:17.699820 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:17.699972 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:17.701292 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:17.702327 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:17.703529 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:17.704120 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:17.704232 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:17.708235 (+ 4003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:17.708246 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:17.708258 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:17.708293 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:17.710897 (+ 2604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:17.711486 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:17.711493 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:17.711495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:17.711612 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:17.711908 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:17.711911 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:17.711913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:17.712230 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:17.712474 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:17.712671 (+ 197us) server_negotiation.cc:299] Negotiation successful
0808 04:04:17.712915 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":325,"threads_started":1}
W20260808 04:04:17.714386 10376 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:35116
W20260808 04:04:17.715356 5452 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 10214: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 04:04:17.715528 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 10214
I20260808 04:04:17.732720 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 9724
I20260808 04:04:17.733753 9814 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:04:17.833586 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 9724
2026-08-08T04:04:17Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3630 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:04:17 dist-test-slave-1l3l krb5kdc[10407](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:04:17 dist-test-slave-1l3l krb5kdc[10407](info): set up 2 sockets
Aug 08 04:04:17 dist-test-slave-1l3l krb5kdc[10407](info): commencing operation
krb5kdc: starting...
W20260808 04:04:19.005509 5452 mini_kdc.cc:121] Time spent starting KDC: real 1.128s user 0.001s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/krb5kdc/test-user.keytab.
Aug 08 04:04:19 dist-test-slave-1l3l krb5kdc[10407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161859, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:04:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:04:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.83.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab.
Entry for principal kudu/127.5.83.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab.
I20260808 04:04:19.200606 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:34151
--webserver_interface=127.5.83.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab
--principal=kudu/127.5.83.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:39653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.62:34151
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:04:19.528728 10422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:04:19.529014 10422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:04:19.529165 10422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:04:19.536216 10422 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:04:19.536332 10422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:04:19.536410 10422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:04:19.536489 10422 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:04:19.536550 10422 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:04:19.545814 10422 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:39653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.62:34151
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab
--principal=kudu/127.5.83.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:34151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:04:19.547037 10422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:04:19.548332 10422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:04:19.559257 10434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:19.559625 10430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:19.559674 10431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:04:19.560333 10422 server_base.cc:1034] running on GCE node
I20260808 04:04:19.561096 10422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:04:19.562472 10422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:04:19.563674 10422 hybrid_clock.cc:648] HybridClock initialized: now 1786161859563619 us; error 43 us; skew 500 ppm
I20260808 04:04:19.564028 10422 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:04:19 dist-test-slave-1l3l krb5kdc[10407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161859, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:04:19.571120 10422 init.cc:377] Logged in from keytab as kudu/127.5.83.62@KRBTEST.COM (short username kudu)
I20260808 04:04:19.572552 10422 webserver.cc:460] Webserver started at http://127.5.83.62:36481/ using document root <none> and password file <none>
I20260808 04:04:19.573196 10422 fs_manager.cc:359] Metadata directory not provided
I20260808 04:04:19.573312 10422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:04:19.573611 10422 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:04:19.576583 10422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/minicluster-data/master-0/data/instance:
uuid: "f1edaafc233f417ab8e43e230bccc547"
format_stamp: "Formatted at 2026-08-08 04:04:19 on dist-test-slave-1l3l"
server_key: "62f1e170cc89196032369604111475b0"
server_key_iv: "9d777a3caee1eda6d41f31f40750cf6b"
server_key_version: "encryptionkey@0"
I20260808 04:04:19.577344 10422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/minicluster-data/master-0/wal/instance:
uuid: "f1edaafc233f417ab8e43e230bccc547"
format_stamp: "Formatted at 2026-08-08 04:04:19 on dist-test-slave-1l3l"
server_key: "62f1e170cc89196032369604111475b0"
server_key_iv: "9d777a3caee1eda6d41f31f40750cf6b"
server_key_version: "encryptionkey@0"
I20260808 04:04:19.582182 10422 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260808 04:04:19.585462 10443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:19.586694 10422 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 04:04:19.586844 10422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/minicluster-data/master-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/minicluster-data/master-0/wal
uuid: "f1edaafc233f417ab8e43e230bccc547"
format_stamp: "Formatted at 2026-08-08 04:04:19 on dist-test-slave-1l3l"
server_key: "62f1e170cc89196032369604111475b0"
server_key_iv: "9d777a3caee1eda6d41f31f40750cf6b"
server_key_version: "encryptionkey@0"
I20260808 04:04:19.587030 10422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:04:19.612967 10422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:04:19.617053 10422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:04:19.617304 10422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:04:19.634699 10422 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.62:34151
I20260808 04:04:19.634720 10513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.62:34151 every 8 connection(s)
I20260808 04:04:19.636101 10422 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
I20260808 04:04:19.637140 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 10422
I20260808 04:04:19.637368 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/minicluster-data/master-0/wal/instance
I20260808 04:04:19.638000 5452 external_mini_cluster.cc:1428] Setting key 48dbcb5ae6a3334a181cbc2e3b3e5f9a
I20260808 04:04:19.640501 10514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 08 04:04:19 dist-test-slave-1l3l krb5kdc[10407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161859, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:04:19.653512 10514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1edaafc233f417ab8e43e230bccc547: Bootstrap starting.
I20260808 04:04:19.656978 10514 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f1edaafc233f417ab8e43e230bccc547: Neither blocks nor log segments found. Creating new log.
I20260808 04:04:19.658002 10514 log.cc:824] T 00000000000000000000000000000000 P f1edaafc233f417ab8e43e230bccc547: Log is configured to *not* fsync() on all Append() calls
I20260808 04:04:19.660784 10514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1edaafc233f417ab8e43e230bccc547: No bootstrap required, opened a new log
I20260808 04:04:19.660873 10521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:19.639968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48232 (local address 127.5.83.62:34151)
0808 04:04:19.640856 (+ 888us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:19.640872 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:19.640925 (+ 53us) server_negotiation.cc:407] Connection header received
0808 04:04:19.643169 (+ 2244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:19.643196 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:19.643691 (+ 495us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:19.644170 (+ 479us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:19.645451 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:19.647480 (+ 2029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:19.648810 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:19.649837 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:19.650016 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:19.654095 (+ 4079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:19.654103 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:19.654116 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:19.654150 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:19.657078 (+ 2928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:19.657740 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:19.657746 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:19.657752 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:19.657933 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:19.658315 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:19.658318 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:19.658320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:19.658840 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:19.659135 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:19.659567 (+ 432us) server_negotiation.cc:299] Negotiation successful
0808 04:04:19.659976 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":363,"threads_started":1}
I20260808 04:04:19.668753 10514 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1edaafc233f417ab8e43e230bccc547 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1edaafc233f417ab8e43e230bccc547" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 34151 } }
I20260808 04:04:19.669106 10514 raft_consensus.cc:374] T 00000000000000000000000000000000 P f1edaafc233f417ab8e43e230bccc547 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:04:19.669196 10514 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1edaafc233f417ab8e43e230bccc547 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1edaafc233f417ab8e43e230bccc547, State: Initialized, Role: FOLLOWER
I20260808 04:04:19.669760 10514 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1edaafc233f417ab8e43e230bccc547 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1edaafc233f417ab8e43e230bccc547" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 34151 } }
I20260808 04:04:19.669987 10514 raft_consensus.cc:388] T 00000000000000000000000000000000 P f1edaafc233f417ab8e43e230bccc547 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:04:19.670114 10514 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1edaafc233f417ab8e43e230bccc547 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:04:19.670272 10514 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1edaafc233f417ab8e43e230bccc547 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:04:19.672457 10514 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1edaafc233f417ab8e43e230bccc547 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1edaafc233f417ab8e43e230bccc547" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 34151 } }
I20260808 04:04:19.672946 10514 leader_election.cc:302] T 00000000000000000000000000000000 P f1edaafc233f417ab8e43e230bccc547 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f1edaafc233f417ab8e43e230bccc547; no voters:
I20260808 04:04:19.673503 10514 leader_election.cc:288] T 00000000000000000000000000000000 P f1edaafc233f417ab8e43e230bccc547 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:04:19.673635 10523 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1edaafc233f417ab8e43e230bccc547 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:04:19.674978 10523 raft_consensus.cc:686] T 00000000000000000000000000000000 P f1edaafc233f417ab8e43e230bccc547 [term 1 LEADER]: Becoming Leader. State: Replica: f1edaafc233f417ab8e43e230bccc547, State: Running, Role: LEADER
I20260808 04:04:19.675554 10514 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1edaafc233f417ab8e43e230bccc547 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:04:19.675491 10523 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1edaafc233f417ab8e43e230bccc547 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1edaafc233f417ab8e43e230bccc547" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 34151 } }
I20260808 04:04:19.678143 10527 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1edaafc233f417ab8e43e230bccc547 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1edaafc233f417ab8e43e230bccc547. Latest consensus state: current_term: 1 leader_uuid: "f1edaafc233f417ab8e43e230bccc547" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1edaafc233f417ab8e43e230bccc547" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 34151 } } }
I20260808 04:04:19.678609 10527 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1edaafc233f417ab8e43e230bccc547 [sys.catalog]: This master's current role is: LEADER
I20260808 04:04:19.678256 10526 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1edaafc233f417ab8e43e230bccc547 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f1edaafc233f417ab8e43e230bccc547" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1edaafc233f417ab8e43e230bccc547" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 34151 } } }
I20260808 04:04:19.679095 10526 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1edaafc233f417ab8e43e230bccc547 [sys.catalog]: This master's current role is: LEADER
I20260808 04:04:19.679446 10535 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:04:19.684749 10535 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:04:19.696727 10535 catalog_manager.cc:1347] Generated new cluster ID: 7c3a78c19e104ee6b6c06b8b86eb682e
I20260808 04:04:19.696856 10535 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:04:19.711076 10535 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:04:19.712330 10535 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:04:19.718559 10535 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f1edaafc233f417ab8e43e230bccc547: Generated new TSK 0
I20260808 04:04:19.719300 10535 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.83.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.1.keytab.
Entry for principal kudu/127.5.83.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.1.keytab.
I20260808 04:04:19.774945 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.1:0
--local_ip_for_outbound_sockets=127.5.83.1
--webserver_interface=127.5.83.1
--webserver_port=0
--tserver_master_addrs=127.5.83.62:34151
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.1.keytab
--principal=kudu/127.5.83.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:39653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:04:20.091063 10555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:04:20.091362 10555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:04:20.091471 10555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:04:20.098469 10555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:04:20.098567 10555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:04:20.098681 10555 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.1
I20260808 04:04:20.107936 10555 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:39653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.1.keytab
--principal=kudu/127.5.83.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.83.1
--webserver_port=0
--tserver_master_addrs=127.5.83.62:34151
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.1
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:04:20.109064 10555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:04:20.110275 10555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:04:20.122637 10562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:20.122660 10566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:20.122763 10564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:04:20.124425 10555 server_base.cc:1034] running on GCE node
I20260808 04:04:20.124984 10555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:04:20.125814 10555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:04:20.127043 10555 hybrid_clock.cc:648] HybridClock initialized: now 1786161860126982 us; error 51 us; skew 500 ppm
I20260808 04:04:20.127342 10555 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:04:20 dist-test-slave-1l3l krb5kdc[10407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161860, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:04:20.130502 10555 init.cc:377] Logged in from keytab as kudu/127.5.83.1@KRBTEST.COM (short username kudu)
I20260808 04:04:20.131886 10555 webserver.cc:460] Webserver started at http://127.5.83.1:45157/ using document root <none> and password file <none>
I20260808 04:04:20.132527 10555 fs_manager.cc:359] Metadata directory not provided
I20260808 04:04:20.132624 10555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:04:20.132876 10555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:04:20.135663 10555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/minicluster-data/ts-0/data/instance:
uuid: "723a17f91c8e41fb82c4de828bec4f1c"
format_stamp: "Formatted at 2026-08-08 04:04:20 on dist-test-slave-1l3l"
server_key: "75bf26164b7a0b4b89c26507bd41b211"
server_key_iv: "08e4e063f15f6d30c4547ea834c00b52"
server_key_version: "encryptionkey@0"
I20260808 04:04:20.136416 10555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/minicluster-data/ts-0/wal/instance:
uuid: "723a17f91c8e41fb82c4de828bec4f1c"
format_stamp: "Formatted at 2026-08-08 04:04:20 on dist-test-slave-1l3l"
server_key: "75bf26164b7a0b4b89c26507bd41b211"
server_key_iv: "08e4e063f15f6d30c4547ea834c00b52"
server_key_version: "encryptionkey@0"
I20260808 04:04:20.141155 10555 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 04:04:20.144346 10574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:20.145556 10555 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:04:20.145687 10555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/minicluster-data/ts-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/minicluster-data/ts-0/wal
uuid: "723a17f91c8e41fb82c4de828bec4f1c"
format_stamp: "Formatted at 2026-08-08 04:04:20 on dist-test-slave-1l3l"
server_key: "75bf26164b7a0b4b89c26507bd41b211"
server_key_iv: "08e4e063f15f6d30c4547ea834c00b52"
server_key_version: "encryptionkey@0"
I20260808 04:04:20.145841 10555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:04:20.168085 10555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:04:20.171624 10555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:04:20.171891 10555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:04:20.172758 10555 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:04:20.173988 10555 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:04:20.174063 10555 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:20.174149 10555 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:04:20.174191 10555 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:20.204198 10555 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.1:45087
I20260808 04:04:20.204233 10724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.1:45087 every 8 connection(s)
I20260808 04:04:20.205407 10555 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
I20260808 04:04:20.214524 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 10555
I20260808 04:04:20.214715 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/minicluster-data/ts-0/wal/instance
I20260808 04:04:20.215483 5452 external_mini_cluster.cc:1428] Setting key 5f950c3c61502161a3e84f2d976b983b
Aug 08 04:04:20 dist-test-slave-1l3l krb5kdc[10407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161860, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.1@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:04:20.225550 10729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:20.207719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.1:57173 (local address 127.5.83.62:34151)
0808 04:04:20.208150 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:20.208154 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:20.209082 (+ 928us) server_negotiation.cc:407] Connection header received
0808 04:04:20.211001 (+ 1919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:20.211007 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:20.211095 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:20.211192 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:20.212723 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:20.213477 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:20.215065 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:20.215520 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:20.215633 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:20.219547 (+ 3914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:20.219552 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:20.219555 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:20.219588 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:20.222385 (+ 2797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:20.223183 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:20.223187 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:20.223189 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:20.223305 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:20.223761 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:20.223765 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:20.223767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:20.224042 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:20.224154 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:20.224945 (+ 791us) server_negotiation.cc:299] Negotiation successful
0808 04:04:20.225173 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":230,"threads_started":1}
I20260808 04:04:20.226459 10730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:20.208175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:34151 (local address 127.5.83.1:57173)
0808 04:04:20.208876 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:20.208923 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:20.210722 (+ 1799us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:20.211434 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:20.211443 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:20.211995 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:04:20.212554 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:20.212572 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:20.213622 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:20.213626 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:20.214927 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:20.214941 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:20.215777 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:20.215787 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:20.216105 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:20.216739 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:04:20.216761 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:04:20.219295 (+ 2534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:04:20.222521 (+ 3226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:20.222530 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:20.222546 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:20.223080 (+ 534us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:20.223396 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:20.223401 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:20.223404 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:20.223629 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:20.224159 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:04:20.224189 (+ 30us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:04:20.224590 (+ 401us) client_negotiation.cc:769] Sending connection context
0808 04:04:20.224874 (+ 284us) client_negotiation.cc:240] Negotiation successful
0808 04:04:20.225328 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":291,"threads_started":1}
I20260808 04:04:20.227983 10726 heartbeater.cc:347] Connected to a master server at 127.5.83.62:34151
I20260808 04:04:20.228353 10726 heartbeater.cc:464] Registering TS with master...
I20260808 04:04:20.229182 10726 heartbeater.cc:511] Master 127.5.83.62:34151 requested a full tablet report, sending...
I20260808 04:04:20.231251 10461 ts_manager.cc:194] Registered new tserver with Master: 723a17f91c8e41fb82c4de828bec4f1c (127.5.83.1:45087)
I20260808 04:04:20.232677 10461 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.1@KRBTEST.COM'} at 127.5.83.1:57173
WARNING: no policy specified for kudu/127.5.83.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.2.keytab.
Entry for principal kudu/127.5.83.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.2.keytab.
I20260808 04:04:20.262413 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.2:0
--local_ip_for_outbound_sockets=127.5.83.2
--webserver_interface=127.5.83.2
--webserver_port=0
--tserver_master_addrs=127.5.83.62:34151
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.2.keytab
--principal=kudu/127.5.83.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:39653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:04:20.575438 10734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:04:20.575736 10734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:04:20.575853 10734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:04:20.582999 10734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:04:20.583094 10734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:04:20.583207 10734 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.2
I20260808 04:04:20.592696 10734 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:39653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.2.keytab
--principal=kudu/127.5.83.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.83.2
--webserver_port=0
--tserver_master_addrs=127.5.83.62:34151
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.2
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:04:20.593827 10734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:04:20.595064 10734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:04:20.607442 10744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:20.607406 10742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:20.607905 10741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:04:20.608361 10734 server_base.cc:1034] running on GCE node
I20260808 04:04:20.608918 10734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:04:20.609819 10734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:04:20.611016 10734 hybrid_clock.cc:648] HybridClock initialized: now 1786161860610970 us; error 39 us; skew 500 ppm
I20260808 04:04:20.611279 10734 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:04:20 dist-test-slave-1l3l krb5kdc[10407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161860, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:04:20.614203 10734 init.cc:377] Logged in from keytab as kudu/127.5.83.2@KRBTEST.COM (short username kudu)
I20260808 04:04:20.615526 10734 webserver.cc:460] Webserver started at http://127.5.83.2:34941/ using document root <none> and password file <none>
I20260808 04:04:20.616171 10734 fs_manager.cc:359] Metadata directory not provided
I20260808 04:04:20.616279 10734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:04:20.616575 10734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:04:20.619330 10734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/minicluster-data/ts-1/data/instance:
uuid: "356fcc0e8ed64ac4bfd10b879ff90900"
format_stamp: "Formatted at 2026-08-08 04:04:20 on dist-test-slave-1l3l"
server_key: "41642b996ce0bf76d77a34aa94b85a4b"
server_key_iv: "a2d2e74ad8deb774319151257c25bea4"
server_key_version: "encryptionkey@0"
I20260808 04:04:20.620097 10734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/minicluster-data/ts-1/wal/instance:
uuid: "356fcc0e8ed64ac4bfd10b879ff90900"
format_stamp: "Formatted at 2026-08-08 04:04:20 on dist-test-slave-1l3l"
server_key: "41642b996ce0bf76d77a34aa94b85a4b"
server_key_iv: "a2d2e74ad8deb774319151257c25bea4"
server_key_version: "encryptionkey@0"
I20260808 04:04:20.624993 10734 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260808 04:04:20.628276 10751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:20.629508 10734 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:04:20.629660 10734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/minicluster-data/ts-1/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/minicluster-data/ts-1/wal
uuid: "356fcc0e8ed64ac4bfd10b879ff90900"
format_stamp: "Formatted at 2026-08-08 04:04:20 on dist-test-slave-1l3l"
server_key: "41642b996ce0bf76d77a34aa94b85a4b"
server_key_iv: "a2d2e74ad8deb774319151257c25bea4"
server_key_version: "encryptionkey@0"
I20260808 04:04:20.629848 10734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:04:20.664667 10734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:04:20.668918 10734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:04:20.669193 10734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:04:20.670156 10734 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:04:20.671468 10734 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:04:20.671547 10734 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:20.671681 10734 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:04:20.671759 10734 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260808 04:04:20.701730 10734 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.2:34325
I20260808 04:04:20.701778 10907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.2:34325 every 8 connection(s)
I20260808 04:04:20.702947 10734 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
I20260808 04:04:20.708489 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 10734
I20260808 04:04:20.708678 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/minicluster-data/ts-1/wal/instance
I20260808 04:04:20.709169 5452 external_mini_cluster.cc:1428] Setting key 6b4e01b346ca955cfd501e80be927061
Aug 08 04:04:20 dist-test-slave-1l3l krb5kdc[10407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161860, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.2@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:04:20.723153 10729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:20.705460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.2:34571 (local address 127.5.83.62:34151)
0808 04:04:20.705679 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:20.705685 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:20.706787 (+ 1102us) server_negotiation.cc:407] Connection header received
0808 04:04:20.708619 (+ 1832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:20.708625 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:20.708719 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:20.708834 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:20.710397 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:20.711349 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:20.712936 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:20.713484 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:20.713601 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:20.717103 (+ 3502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:20.717108 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:20.717111 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:20.717142 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:20.720194 (+ 3052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:20.720858 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:20.720863 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:20.720866 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:20.721020 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:20.721420 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:20.721425 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:20.721428 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:20.721771 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:20.721917 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:20.722554 (+ 637us) server_negotiation.cc:299] Negotiation successful
0808 04:04:20.722768 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260808 04:04:20.723969 10910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:20.705938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:34151 (local address 127.5.83.2:34571)
0808 04:04:20.706623 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:20.706668 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:20.708362 (+ 1694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:20.709023 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:20.709030 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:20.709581 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:04:20.710233 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:20.710250 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:20.711510 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:20.711514 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:20.712789 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:20.712802 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:20.713665 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:20.713669 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:20.713854 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:20.714467 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:04:20.714503 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:04:20.716901 (+ 2398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:04:20.720318 (+ 3417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:20.720324 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:20.720335 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:20.720730 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:20.721126 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:20.721129 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:20.721132 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:20.721298 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:20.721896 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:04:20.721902 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:04:20.722191 (+ 289us) client_negotiation.cc:769] Sending connection context
0808 04:04:20.722415 (+ 224us) client_negotiation.cc:240] Negotiation successful
0808 04:04:20.722762 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":272,"threads_started":1}
I20260808 04:04:20.725271 10908 heartbeater.cc:347] Connected to a master server at 127.5.83.62:34151
I20260808 04:04:20.725610 10908 heartbeater.cc:464] Registering TS with master...
I20260808 04:04:20.726431 10908 heartbeater.cc:511] Master 127.5.83.62:34151 requested a full tablet report, sending...
I20260808 04:04:20.728073 10461 ts_manager.cc:194] Registered new tserver with Master: 356fcc0e8ed64ac4bfd10b879ff90900 (127.5.83.2:34325)
I20260808 04:04:20.728811 10461 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.2@KRBTEST.COM'} at 127.5.83.2:34571
WARNING: no policy specified for kudu/127.5.83.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.3.keytab.
Entry for principal kudu/127.5.83.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.3.keytab.
I20260808 04:04:20.756541 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.3:0
--local_ip_for_outbound_sockets=127.5.83.3
--webserver_interface=127.5.83.3
--webserver_port=0
--tserver_master_addrs=127.5.83.62:34151
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.3.keytab
--principal=kudu/127.5.83.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:39653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:04:21.075326 10913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:04:21.075632 10913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:04:21.075803 10913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:04:21.083032 10913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:04:21.083154 10913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:04:21.083308 10913 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.3
I20260808 04:04:21.093071 10913 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:39653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.3.keytab
--principal=kudu/127.5.83.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.83.3
--webserver_port=0
--tserver_master_addrs=127.5.83.62:34151
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.3
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:04:21.094352 10913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:04:21.095685 10913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:04:21.107839 10920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:21.107900 10923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:21.108021 10921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:04:21.108629 10913 server_base.cc:1034] running on GCE node
I20260808 04:04:21.109150 10913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:04:21.109969 10913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:04:21.111168 10913 hybrid_clock.cc:648] HybridClock initialized: now 1786161861111123 us; error 34 us; skew 500 ppm
I20260808 04:04:21.111464 10913 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:04:21 dist-test-slave-1l3l krb5kdc[10407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161861, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:04:21.114532 10913 init.cc:377] Logged in from keytab as kudu/127.5.83.3@KRBTEST.COM (short username kudu)
I20260808 04:04:21.116003 10913 webserver.cc:460] Webserver started at http://127.5.83.3:39109/ using document root <none> and password file <none>
I20260808 04:04:21.116631 10913 fs_manager.cc:359] Metadata directory not provided
I20260808 04:04:21.116748 10913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:04:21.117038 10913 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:04:21.119872 10913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/minicluster-data/ts-2/data/instance:
uuid: "51929cd977424707becd842d49661f2e"
format_stamp: "Formatted at 2026-08-08 04:04:21 on dist-test-slave-1l3l"
server_key: "91f1cb45dfb82bc7d19520104fbd6526"
server_key_iv: "5f7298cae1833b59b648db98756b64f6"
server_key_version: "encryptionkey@0"
I20260808 04:04:21.120609 10913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/minicluster-data/ts-2/wal/instance:
uuid: "51929cd977424707becd842d49661f2e"
format_stamp: "Formatted at 2026-08-08 04:04:21 on dist-test-slave-1l3l"
server_key: "91f1cb45dfb82bc7d19520104fbd6526"
server_key_iv: "5f7298cae1833b59b648db98756b64f6"
server_key_version: "encryptionkey@0"
I20260808 04:04:21.125378 10913 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 04:04:21.128605 10931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:21.129801 10913 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:04:21.129943 10913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/minicluster-data/ts-2/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/minicluster-data/ts-2/wal
uuid: "51929cd977424707becd842d49661f2e"
format_stamp: "Formatted at 2026-08-08 04:04:21 on dist-test-slave-1l3l"
server_key: "91f1cb45dfb82bc7d19520104fbd6526"
server_key_iv: "5f7298cae1833b59b648db98756b64f6"
server_key_version: "encryptionkey@0"
I20260808 04:04:21.130118 10913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:04:21.144647 10913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:04:21.148228 10913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:04:21.148497 10913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:04:21.149379 10913 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:04:21.150658 10913 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:04:21.150734 10913 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:21.150851 10913 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:04:21.150909 10913 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:21.181427 10913 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.3:44047
I20260808 04:04:21.181474 11077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.3:44047 every 8 connection(s)
I20260808 04:04:21.182660 10913 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
I20260808 04:04:21.183629 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 10913
I20260808 04:04:21.183861 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161826605655-5452-0/minicluster-data/ts-2/wal/instance
I20260808 04:04:21.184280 5452 external_mini_cluster.cc:1428] Setting key bbdbe16ff59201edfbbf0a3a65974f0c
Aug 08 04:04:21 dist-test-slave-1l3l krb5kdc[10407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161861, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.3@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:04:21.201403 10729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:21.185139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.3:43527 (local address 127.5.83.62:34151)
0808 04:04:21.185340 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:21.185345 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:21.186612 (+ 1267us) server_negotiation.cc:407] Connection header received
0808 04:04:21.188460 (+ 1848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:21.188465 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:21.188542 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:21.188624 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:21.190034 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:21.190879 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:21.192418 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:21.192838 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:21.192950 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:21.196149 (+ 3199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:21.196154 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:21.196157 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:21.196190 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:21.198767 (+ 2577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:21.199562 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:21.199566 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:21.199568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:21.199692 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:21.200072 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:21.200075 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:21.200077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:21.200324 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:21.200428 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:21.201022 (+ 594us) server_negotiation.cc:299] Negotiation successful
0808 04:04:21.201176 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 04:04:21.202185 11080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:21.185686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:34151 (local address 127.5.83.3:43527)
0808 04:04:21.186470 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:21.186507 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:21.188211 (+ 1704us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:21.188849 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:21.188856 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:21.189334 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:04:21.189876 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:21.189895 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:21.191021 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:21.191025 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:21.192286 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:21.192299 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:21.193054 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:21.193062 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:21.193282 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:21.193870 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:04:21.193891 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:04:21.195977 (+ 2086us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:04:21.198920 (+ 2943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:21.198928 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:21.198942 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:21.199456 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:21.199800 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:21.199803 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:21.199806 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:21.199988 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:21.200412 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:04:21.200417 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:04:21.200685 (+ 268us) client_negotiation.cc:769] Sending connection context
0808 04:04:21.200892 (+ 207us) client_negotiation.cc:240] Negotiation successful
0808 04:04:21.201235 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":290,"threads_started":1}
I20260808 04:04:21.203316 11078 heartbeater.cc:347] Connected to a master server at 127.5.83.62:34151
I20260808 04:04:21.203626 11078 heartbeater.cc:464] Registering TS with master...
I20260808 04:04:21.204526 11078 heartbeater.cc:511] Master 127.5.83.62:34151 requested a full tablet report, sending...
I20260808 04:04:21.206074 10461 ts_manager.cc:194] Registered new tserver with Master: 51929cd977424707becd842d49661f2e (127.5.83.3:44047)
I20260808 04:04:21.207152 10461 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.3@KRBTEST.COM'} at 127.5.83.3:43527
I20260808 04:04:21.211293 5452 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:04:21.236138 10726 heartbeater.cc:503] Master 127.5.83.62:34151 was elected leader, sending a full tablet report...
Aug 08 04:04:21 dist-test-slave-1l3l krb5kdc[10407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161861, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 04:04:21 dist-test-slave-1l3l krb5kdc[10407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161861, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.1@KRBTEST.COM
I20260808 04:04:21.261722 11087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:21.247137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41120 (local address 127.5.83.1:45087)
0808 04:04:21.247822 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:21.247831 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:21.247858 (+ 27us) server_negotiation.cc:407] Connection header received
0808 04:04:21.247916 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:21.247923 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:21.248121 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:21.248280 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:21.249666 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:21.250740 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:21.251995 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:21.252565 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:21.252680 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:21.256414 (+ 3734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:21.256425 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:21.256438 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:21.256470 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:21.259067 (+ 2597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:21.259672 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:21.259678 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:21.259681 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:21.259847 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:21.260202 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:21.260206 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:21.260208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:21.260545 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:21.260822 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:21.261146 (+ 324us) server_negotiation.cc:299] Negotiation successful
0808 04:04:21.261403 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":328,"threads_started":1}
Aug 08 04:04:21 dist-test-slave-1l3l krb5kdc[10407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161861, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:04:21.278777 10729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:21.265837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48244 (local address 127.5.83.62:34151)
0808 04:04:21.266048 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:21.266054 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:21.266219 (+ 165us) server_negotiation.cc:407] Connection header received
0808 04:04:21.266342 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:21.266346 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:21.266422 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:21.266501 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:21.267745 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:21.268465 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:21.269614 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:21.270036 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:21.270126 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:21.273732 (+ 3606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:21.273737 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:21.273740 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:21.273770 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:21.276199 (+ 2429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:21.276824 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:21.276828 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:21.276830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:21.276952 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:21.277421 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:21.277426 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:21.277430 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:21.277736 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:21.277975 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:21.278287 (+ 312us) server_negotiation.cc:299] Negotiation successful
0808 04:04:21.278497 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260808 04:04:21.279372 10461 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48244
I20260808 04:04:21.280315 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 10555
I20260808 04:04:21.291561 11087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:21.281070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41132 (local address 127.5.83.1:45087)
0808 04:04:21.281222 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:21.281226 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:21.281496 (+ 270us) server_negotiation.cc:407] Connection header received
0808 04:04:21.281617 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:21.281620 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:21.281706 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:21.281782 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:21.282833 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:21.283524 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:21.284688 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:21.285082 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:21.285174 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:21.287186 (+ 2012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:21.287191 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:21.287194 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:21.287223 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:21.289531 (+ 2308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:21.290124 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:21.290128 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:21.290130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:21.290244 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:21.290532 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:21.290535 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:21.290537 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:21.290787 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:21.290960 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:21.291179 (+ 219us) server_negotiation.cc:299] Negotiation successful
0808 04:04:21.291334 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 04:04:21.292133 10717 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:04:21.375164 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 10555
I20260808 04:04:21.396545 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 10734
Aug 08 04:04:21 dist-test-slave-1l3l krb5kdc[10407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161861, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.2@KRBTEST.COM
I20260808 04:04:21.411908 11093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:21.397408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52770 (local address 127.5.83.2:34325)
0808 04:04:21.398002 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:21.398009 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:21.398036 (+ 27us) server_negotiation.cc:407] Connection header received
0808 04:04:21.398100 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:21.398106 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:21.398308 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:21.398448 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:21.399571 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:21.400612 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:21.401773 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:21.402306 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:21.402399 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:21.406367 (+ 3968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:21.406380 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:21.406394 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:21.406436 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:21.409143 (+ 2707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:21.409791 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:21.409798 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:21.409800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:21.409933 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:21.410305 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:21.410309 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:21.410311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:21.410684 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:21.410976 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:21.411260 (+ 284us) server_negotiation.cc:299] Negotiation successful
0808 04:04:21.411520 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":299,"threads_started":1}
I20260808 04:04:21.413125 10902 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:04:21.493783 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 10734
I20260808 04:04:21.516575 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 10913
Aug 08 04:04:21 dist-test-slave-1l3l krb5kdc[10407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161861, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.3@KRBTEST.COM
I20260808 04:04:21.535429 11095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:21.517877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59744 (local address 127.5.83.3:44047)
0808 04:04:21.518592 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:21.518598 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:21.518629 (+ 31us) server_negotiation.cc:407] Connection header received
0808 04:04:21.518687 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:21.518693 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:21.518895 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:21.519028 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:21.520803 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:21.522460 (+ 1657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:21.524666 (+ 2206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:21.525667 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:21.525810 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:21.530192 (+ 4382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:21.530204 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:21.530218 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:21.530252 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:21.532995 (+ 2743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:21.533642 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:21.533648 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:21.533651 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:21.533770 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:21.534056 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:21.534059 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:21.534061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:21.534378 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:21.534626 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:21.534922 (+ 296us) server_negotiation.cc:299] Negotiation successful
0808 04:04:21.535119 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":381,"threads_started":1}
I20260808 04:04:21.536696 11071 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:04:21.616948 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 10913
I20260808 04:04:21.633944 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 10422
I20260808 04:04:21.635353 10507 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:04:21.738696 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 10422
2026-08-08T04:04:21Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3905 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:04:21 dist-test-slave-1l3l krb5kdc[11100](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:04:21 dist-test-slave-1l3l krb5kdc[11100](info): set up 2 sockets
Aug 08 04:04:21 dist-test-slave-1l3l krb5kdc[11100](info): commencing operation
krb5kdc: starting...
W20260808 04:04:22.906160 5452 mini_kdc.cc:121] Time spent starting KDC: real 1.123s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/krb5kdc/test-user.keytab.
Aug 08 04:04:23 dist-test-slave-1l3l krb5kdc[11100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:04:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:04:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.83.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab.
Entry for principal kudu/127.5.83.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab.
I20260808 04:04:23.098660 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:33649
--webserver_interface=127.5.83.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab
--principal=kudu/127.5.83.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:33749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.62:33649
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:04:23.412536 11120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:04:23.412807 11120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:04:23.412930 11120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:04:23.419875 11120 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:04:23.419971 11120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:04:23.420045 11120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:04:23.420115 11120 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:04:23.420166 11120 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:04:23.429600 11120 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:33749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.62:33649
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab
--principal=kudu/127.5.83.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:33649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:04:23.430778 11120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:04:23.432067 11120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:04:23.442909 11125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:23.443012 11126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:23.443038 11132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:04:23.444242 11120 server_base.cc:1034] running on GCE node
I20260808 04:04:23.444980 11120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:04:23.446305 11120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:04:23.447537 11120 hybrid_clock.cc:648] HybridClock initialized: now 1786161863447486 us; error 45 us; skew 500 ppm
I20260808 04:04:23.447911 11120 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:04:23 dist-test-slave-1l3l krb5kdc[11100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161863, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:04:23.450887 11120 init.cc:377] Logged in from keytab as kudu/127.5.83.62@KRBTEST.COM (short username kudu)
I20260808 04:04:23.452044 11120 fs_manager.cc:359] Metadata directory not provided
I20260808 04:04:23.452152 11120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:04:23.452462 11120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:04:23.455291 11120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/minicluster-data/master-0/data/instance:
uuid: "b81776e15e0c4e3cb5c378df1fa84c24"
format_stamp: "Formatted at 2026-08-08 04:04:23 on dist-test-slave-1l3l"
server_key: "f32497f7e12df12129b63b84ecbaaefc"
server_key_iv: "f7e95af0a20918cf6e78509ed3bcc101"
server_key_version: "encryptionkey@0"
I20260808 04:04:23.456131 11120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/minicluster-data/master-0/wal/instance:
uuid: "b81776e15e0c4e3cb5c378df1fa84c24"
format_stamp: "Formatted at 2026-08-08 04:04:23 on dist-test-slave-1l3l"
server_key: "f32497f7e12df12129b63b84ecbaaefc"
server_key_iv: "f7e95af0a20918cf6e78509ed3bcc101"
server_key_version: "encryptionkey@0"
I20260808 04:04:23.460940 11120 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260808 04:04:23.464449 11139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:23.465665 11120 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:04:23.465816 11120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/minicluster-data/master-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/minicluster-data/master-0/wal
uuid: "b81776e15e0c4e3cb5c378df1fa84c24"
format_stamp: "Formatted at 2026-08-08 04:04:23 on dist-test-slave-1l3l"
server_key: "f32497f7e12df12129b63b84ecbaaefc"
server_key_iv: "f7e95af0a20918cf6e78509ed3bcc101"
server_key_version: "encryptionkey@0"
I20260808 04:04:23.466004 11120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:04:23.480127 11120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:04:23.483691 11120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:04:23.483958 11120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:04:23.500978 11120 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.62:33649
I20260808 04:04:23.500998 11211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.62:33649 every 8 connection(s)
I20260808 04:04:23.501879 11120 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
I20260808 04:04:23.505546 11212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:04:23.505636 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 11120
I20260808 04:04:23.505878 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/minicluster-data/master-0/wal/instance
I20260808 04:04:23.506718 5452 external_mini_cluster.cc:1428] Setting key d90ebdddcb07db0b039c11aec69084d6
I20260808 04:04:23.517575 11212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b81776e15e0c4e3cb5c378df1fa84c24: Bootstrap starting.
Aug 08 04:04:23 dist-test-slave-1l3l krb5kdc[11100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:04:23.520522 11212 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b81776e15e0c4e3cb5c378df1fa84c24: Neither blocks nor log segments found. Creating new log.
I20260808 04:04:23.521485 11212 log.cc:824] T 00000000000000000000000000000000 P b81776e15e0c4e3cb5c378df1fa84c24: Log is configured to *not* fsync() on all Append() calls
I20260808 04:04:23.524256 11212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b81776e15e0c4e3cb5c378df1fa84c24: No bootstrap required, opened a new log
I20260808 04:04:23.528333 11215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:23.508786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40408 (local address 127.5.83.62:33649)
0808 04:04:23.509578 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:23.509589 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:23.509635 (+ 46us) server_negotiation.cc:407] Connection header received
0808 04:04:23.511778 (+ 2143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:23.511800 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:23.512175 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:23.512522 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:23.514010 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:23.515370 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:23.516766 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:23.517411 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:23.517574 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:23.521358 (+ 3784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:23.521366 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:23.521378 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:23.521413 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:23.524427 (+ 3014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:23.525063 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:23.525069 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:23.525075 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:23.525234 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:23.525609 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:23.525612 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:23.525614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:23.526159 (+ 545us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:23.526473 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:23.526893 (+ 420us) server_negotiation.cc:299] Negotiation successful
0808 04:04:23.527266 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":354,"threads_started":1}
I20260808 04:04:23.532282 11212 raft_consensus.cc:348] T 00000000000000000000000000000000 P b81776e15e0c4e3cb5c378df1fa84c24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b81776e15e0c4e3cb5c378df1fa84c24" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 33649 } }
I20260808 04:04:23.532600 11212 raft_consensus.cc:374] T 00000000000000000000000000000000 P b81776e15e0c4e3cb5c378df1fa84c24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:04:23.532691 11212 raft_consensus.cc:729] T 00000000000000000000000000000000 P b81776e15e0c4e3cb5c378df1fa84c24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b81776e15e0c4e3cb5c378df1fa84c24, State: Initialized, Role: FOLLOWER
I20260808 04:04:23.533202 11212 consensus_queue.cc:260] T 00000000000000000000000000000000 P b81776e15e0c4e3cb5c378df1fa84c24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b81776e15e0c4e3cb5c378df1fa84c24" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 33649 } }
I20260808 04:04:23.533403 11212 raft_consensus.cc:388] T 00000000000000000000000000000000 P b81776e15e0c4e3cb5c378df1fa84c24 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:04:23.533527 11212 raft_consensus.cc:482] T 00000000000000000000000000000000 P b81776e15e0c4e3cb5c378df1fa84c24 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:04:23.533660 11212 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b81776e15e0c4e3cb5c378df1fa84c24 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:04:23.535914 11212 raft_consensus.cc:504] T 00000000000000000000000000000000 P b81776e15e0c4e3cb5c378df1fa84c24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b81776e15e0c4e3cb5c378df1fa84c24" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 33649 } }
I20260808 04:04:23.536350 11212 leader_election.cc:302] T 00000000000000000000000000000000 P b81776e15e0c4e3cb5c378df1fa84c24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b81776e15e0c4e3cb5c378df1fa84c24; no voters:
I20260808 04:04:23.536856 11212 leader_election.cc:288] T 00000000000000000000000000000000 P b81776e15e0c4e3cb5c378df1fa84c24 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:04:23.536965 11218 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b81776e15e0c4e3cb5c378df1fa84c24 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:04:23.537563 11218 raft_consensus.cc:686] T 00000000000000000000000000000000 P b81776e15e0c4e3cb5c378df1fa84c24 [term 1 LEADER]: Becoming Leader. State: Replica: b81776e15e0c4e3cb5c378df1fa84c24, State: Running, Role: LEADER
I20260808 04:04:23.538031 11218 consensus_queue.cc:237] T 00000000000000000000000000000000 P b81776e15e0c4e3cb5c378df1fa84c24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b81776e15e0c4e3cb5c378df1fa84c24" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 33649 } }
I20260808 04:04:23.538690 11212 sys_catalog.cc:564] T 00000000000000000000000000000000 P b81776e15e0c4e3cb5c378df1fa84c24 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:04:23.540892 11219 sys_catalog.cc:455] T 00000000000000000000000000000000 P b81776e15e0c4e3cb5c378df1fa84c24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b81776e15e0c4e3cb5c378df1fa84c24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b81776e15e0c4e3cb5c378df1fa84c24" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 33649 } } }
I20260808 04:04:23.540834 11220 sys_catalog.cc:455] T 00000000000000000000000000000000 P b81776e15e0c4e3cb5c378df1fa84c24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b81776e15e0c4e3cb5c378df1fa84c24. Latest consensus state: current_term: 1 leader_uuid: "b81776e15e0c4e3cb5c378df1fa84c24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b81776e15e0c4e3cb5c378df1fa84c24" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 33649 } } }
I20260808 04:04:23.541869 11220 sys_catalog.cc:458] T 00000000000000000000000000000000 P b81776e15e0c4e3cb5c378df1fa84c24 [sys.catalog]: This master's current role is: LEADER
I20260808 04:04:23.542096 11219 sys_catalog.cc:458] T 00000000000000000000000000000000 P b81776e15e0c4e3cb5c378df1fa84c24 [sys.catalog]: This master's current role is: LEADER
I20260808 04:04:23.548033 11227 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:04:23.553056 11227 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:04:23.561244 11227 catalog_manager.cc:1347] Generated new cluster ID: b4b9bd1e099a4971b7e94e5f14426bd3
I20260808 04:04:23.561347 11227 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:04:23.574694 11227 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:04:23.575937 11227 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:04:23.586585 11227 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b81776e15e0c4e3cb5c378df1fa84c24: Generated new TSK 0
I20260808 04:04:23.587303 11227 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.83.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.1.keytab.
Entry for principal kudu/127.5.83.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.1.keytab.
I20260808 04:04:23.639086 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.1:0
--local_ip_for_outbound_sockets=127.5.83.1
--webserver_interface=127.5.83.1
--webserver_port=0
--tserver_master_addrs=127.5.83.62:33649
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.1.keytab
--principal=kudu/127.5.83.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:33749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:04:23.952610 11246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:04:23.952893 11246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:04:23.953030 11246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:04:23.960064 11246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:04:23.960175 11246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:04:23.960335 11246 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.1
I20260808 04:04:23.969599 11246 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:33749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.1.keytab
--principal=kudu/127.5.83.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.83.1
--webserver_port=0
--tserver_master_addrs=127.5.83.62:33649
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.1
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:04:23.970818 11246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:04:23.972055 11246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:04:23.984165 11251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:23.984228 11252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:23.984190 11256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:04:23.985615 11246 server_base.cc:1034] running on GCE node
I20260808 04:04:23.986215 11246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:04:23.987203 11246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:04:23.988409 11246 hybrid_clock.cc:648] HybridClock initialized: now 1786161863988376 us; error 55 us; skew 500 ppm
I20260808 04:04:23.988691 11246 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:04:23 dist-test-slave-1l3l krb5kdc[11100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161863, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:04:23.991747 11246 init.cc:377] Logged in from keytab as kudu/127.5.83.1@KRBTEST.COM (short username kudu)
I20260808 04:04:23.992832 11246 fs_manager.cc:359] Metadata directory not provided
I20260808 04:04:23.992936 11246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:04:23.993240 11246 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:04:23.996073 11246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/minicluster-data/ts-0/data/instance:
uuid: "671a3fb6cf3b489d9ae8d895846b8fd5"
format_stamp: "Formatted at 2026-08-08 04:04:23 on dist-test-slave-1l3l"
server_key: "e2cc52ded11f5d9ec887099ad337c1d6"
server_key_iv: "46d2c79045256a5e5f4acd2b8a0d1db6"
server_key_version: "encryptionkey@0"
I20260808 04:04:23.996857 11246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/minicluster-data/ts-0/wal/instance:
uuid: "671a3fb6cf3b489d9ae8d895846b8fd5"
format_stamp: "Formatted at 2026-08-08 04:04:23 on dist-test-slave-1l3l"
server_key: "e2cc52ded11f5d9ec887099ad337c1d6"
server_key_iv: "46d2c79045256a5e5f4acd2b8a0d1db6"
server_key_version: "encryptionkey@0"
I20260808 04:04:24.001668 11246 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260808 04:04:24.004923 11265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:24.006132 11246 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:04:24.006294 11246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/minicluster-data/ts-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/minicluster-data/ts-0/wal
uuid: "671a3fb6cf3b489d9ae8d895846b8fd5"
format_stamp: "Formatted at 2026-08-08 04:04:23 on dist-test-slave-1l3l"
server_key: "e2cc52ded11f5d9ec887099ad337c1d6"
server_key_iv: "46d2c79045256a5e5f4acd2b8a0d1db6"
server_key_version: "encryptionkey@0"
I20260808 04:04:24.006498 11246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:04:24.021032 11246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:04:24.024991 11246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:04:24.025236 11246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:04:24.026108 11246 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:04:24.027381 11246 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:04:24.027484 11246 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:24.027590 11246 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:04:24.027661 11246 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:24.060195 11246 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.1:40567
I20260808 04:04:24.060248 11418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.1:40567 every 8 connection(s)
I20260808 04:04:24.061023 11246 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
I20260808 04:04:24.069478 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 11246
I20260808 04:04:24.069670 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/minicluster-data/ts-0/wal/instance
I20260808 04:04:24.070286 5452 external_mini_cluster.cc:1428] Setting key c8e678f4fb3577b4e2ad23b0f91debfc
Aug 08 04:04:24 dist-test-slave-1l3l krb5kdc[11100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161863, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.1@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:04:24.081240 11421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:24.063386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.1:40751 (local address 127.5.83.62:33649)
0808 04:04:24.063862 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:24.063867 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:24.064684 (+ 817us) server_negotiation.cc:407] Connection header received
0808 04:04:24.066501 (+ 1817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:24.066507 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:24.066586 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:24.066685 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:24.068194 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:24.068992 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:24.070769 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:24.071308 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:24.071430 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:24.075142 (+ 3712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:24.075147 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:24.075152 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:24.075187 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:24.078329 (+ 3142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:24.079011 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:24.079015 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:24.079018 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:24.079157 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:24.079569 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:24.079573 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:24.079576 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:24.079937 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:24.080070 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:24.080800 (+ 730us) server_negotiation.cc:299] Negotiation successful
0808 04:04:24.080985 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":238,"threads_started":1}
I20260808 04:04:24.082533 11422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:24.063822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:33649 (local address 127.5.83.1:40751)
0808 04:04:24.064529 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:24.064565 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:24.066237 (+ 1672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:24.066935 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:24.066943 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:24.067444 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:04:24.068033 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:24.068051 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:24.069163 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:24.069167 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:24.070638 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:24.070651 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:24.071507 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:24.071512 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:24.071692 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:24.072455 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:04:24.072492 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:04:24.074931 (+ 2439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:04:24.078463 (+ 3532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:24.078470 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:24.078482 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:24.078893 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:24.079249 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:24.079252 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:24.079255 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:24.079462 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:24.080039 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:04:24.080044 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:04:24.080391 (+ 347us) client_negotiation.cc:769] Sending connection context
0808 04:04:24.080703 (+ 312us) client_negotiation.cc:240] Negotiation successful
0808 04:04:24.081197 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":285,"threads_started":1}
I20260808 04:04:24.084004 11419 heartbeater.cc:347] Connected to a master server at 127.5.83.62:33649
I20260808 04:04:24.084270 11419 heartbeater.cc:464] Registering TS with master...
I20260808 04:04:24.084919 11419 heartbeater.cc:511] Master 127.5.83.62:33649 requested a full tablet report, sending...
I20260808 04:04:24.087001 11159 ts_manager.cc:194] Registered new tserver with Master: 671a3fb6cf3b489d9ae8d895846b8fd5 (127.5.83.1:40567)
I20260808 04:04:24.088531 11159 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.1@KRBTEST.COM'} at 127.5.83.1:40751
WARNING: no policy specified for kudu/127.5.83.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.2.keytab.
Entry for principal kudu/127.5.83.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.2.keytab.
I20260808 04:04:24.117041 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.2:0
--local_ip_for_outbound_sockets=127.5.83.2
--webserver_interface=127.5.83.2
--webserver_port=0
--tserver_master_addrs=127.5.83.62:33649
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.2.keytab
--principal=kudu/127.5.83.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:33749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:04:24.431965 11427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:04:24.432224 11427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:04:24.432390 11427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:04:24.439352 11427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:04:24.439471 11427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:04:24.439623 11427 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.2
I20260808 04:04:24.449223 11427 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:33749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.2.keytab
--principal=kudu/127.5.83.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.83.2
--webserver_port=0
--tserver_master_addrs=127.5.83.62:33649
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.2
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:04:24.450470 11427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:04:24.451763 11427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:04:24.463773 11436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:24.463814 11440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:24.463857 11433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:04:24.464773 11427 server_base.cc:1034] running on GCE node
I20260808 04:04:24.465269 11427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:04:24.466101 11427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:04:24.467299 11427 hybrid_clock.cc:648] HybridClock initialized: now 1786161864467247 us; error 45 us; skew 500 ppm
I20260808 04:04:24.467621 11427 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:04:24 dist-test-slave-1l3l krb5kdc[11100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161864, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:04:24.470760 11427 init.cc:377] Logged in from keytab as kudu/127.5.83.2@KRBTEST.COM (short username kudu)
I20260808 04:04:24.471868 11427 fs_manager.cc:359] Metadata directory not provided
I20260808 04:04:24.471964 11427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:04:24.472254 11427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:04:24.474995 11427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/minicluster-data/ts-1/data/instance:
uuid: "70a2055fb6ae45289232a023174c2a78"
format_stamp: "Formatted at 2026-08-08 04:04:24 on dist-test-slave-1l3l"
server_key: "2f2bdfb690dfa06da8bc4d9273a7f409"
server_key_iv: "4b0b2ec2301730ee082578821a7bd5b4"
server_key_version: "encryptionkey@0"
I20260808 04:04:24.475800 11427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/minicluster-data/ts-1/wal/instance:
uuid: "70a2055fb6ae45289232a023174c2a78"
format_stamp: "Formatted at 2026-08-08 04:04:24 on dist-test-slave-1l3l"
server_key: "2f2bdfb690dfa06da8bc4d9273a7f409"
server_key_iv: "4b0b2ec2301730ee082578821a7bd5b4"
server_key_version: "encryptionkey@0"
I20260808 04:04:24.480436 11427 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260808 04:04:24.483565 11449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:24.484789 11427 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 04:04:24.484925 11427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/minicluster-data/ts-1/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/minicluster-data/ts-1/wal
uuid: "70a2055fb6ae45289232a023174c2a78"
format_stamp: "Formatted at 2026-08-08 04:04:24 on dist-test-slave-1l3l"
server_key: "2f2bdfb690dfa06da8bc4d9273a7f409"
server_key_iv: "4b0b2ec2301730ee082578821a7bd5b4"
server_key_version: "encryptionkey@0"
I20260808 04:04:24.485112 11427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:04:24.502027 11427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:04:24.505976 11427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:04:24.506209 11427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:04:24.507120 11427 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:04:24.508424 11427 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:04:24.508507 11427 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:24.508608 11427 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:04:24.508680 11427 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:24.539459 11427 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.2:33045
I20260808 04:04:24.539515 11605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.2:33045 every 8 connection(s)
I20260808 04:04:24.540253 11427 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
I20260808 04:04:24.544107 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 11427
I20260808 04:04:24.544277 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/minicluster-data/ts-1/wal/instance
I20260808 04:04:24.544692 5452 external_mini_cluster.cc:1428] Setting key 0501f59cbaf58a47829667b8598dde23
Aug 08 04:04:24 dist-test-slave-1l3l krb5kdc[11100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161864, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.2@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:04:24.559510 11421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:24.542462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.2:51047 (local address 127.5.83.62:33649)
0808 04:04:24.542621 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:24.542625 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:24.543674 (+ 1049us) server_negotiation.cc:407] Connection header received
0808 04:04:24.545554 (+ 1880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:24.545560 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:24.545642 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:24.545735 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:24.547441 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:24.548509 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:24.550080 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:24.550615 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:24.550734 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:24.554097 (+ 3363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:24.554103 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:24.554108 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:24.554146 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:24.556738 (+ 2592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:24.557513 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:24.557517 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:24.557519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:24.557637 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:24.558069 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:24.558072 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:24.558074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:24.558302 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:24.558412 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:24.559131 (+ 719us) server_negotiation.cc:299] Negotiation successful
0808 04:04:24.559284 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 04:04:24.560880 11608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:24.542836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:33649 (local address 127.5.83.2:51047)
0808 04:04:24.543517 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:24.543555 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:24.545283 (+ 1728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:24.545988 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:24.545998 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:24.546717 (+ 719us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:04:24.547268 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:24.547289 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:24.548641 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:24.548644 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:24.549939 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:24.549951 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:24.550794 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:24.550799 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:24.550985 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:24.551592 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:04:24.551612 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:04:24.553852 (+ 2240us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:04:24.556875 (+ 3023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:24.556883 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:24.556897 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:24.557417 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:24.557749 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:24.557754 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:24.557757 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:24.557974 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:24.558425 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:04:24.558433 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:04:24.558798 (+ 365us) client_negotiation.cc:769] Sending connection context
0808 04:04:24.559080 (+ 282us) client_negotiation.cc:240] Negotiation successful
0808 04:04:24.559614 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":271,"threads_started":1}
I20260808 04:04:24.562260 11606 heartbeater.cc:347] Connected to a master server at 127.5.83.62:33649
I20260808 04:04:24.562572 11606 heartbeater.cc:464] Registering TS with master...
I20260808 04:04:24.563330 11606 heartbeater.cc:511] Master 127.5.83.62:33649 requested a full tablet report, sending...
I20260808 04:04:24.564886 11159 ts_manager.cc:194] Registered new tserver with Master: 70a2055fb6ae45289232a023174c2a78 (127.5.83.2:33045)
I20260808 04:04:24.566020 11159 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.2@KRBTEST.COM'} at 127.5.83.2:51047
WARNING: no policy specified for kudu/127.5.83.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.3.keytab.
Entry for principal kudu/127.5.83.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.3.keytab.
I20260808 04:04:24.594236 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.3:0
--local_ip_for_outbound_sockets=127.5.83.3
--webserver_interface=127.5.83.3
--webserver_port=0
--tserver_master_addrs=127.5.83.62:33649
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.3.keytab
--principal=kudu/127.5.83.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:33749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:04:24.906091 11611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:04:24.906342 11611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:04:24.906438 11611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:04:24.913443 11611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:04:24.913540 11611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:04:24.913658 11611 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.3
I20260808 04:04:24.922832 11611 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:33749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.3.keytab
--principal=kudu/127.5.83.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.83.3
--webserver_port=0
--tserver_master_addrs=127.5.83.62:33649
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.3
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:04:24.923991 11611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:04:24.925194 11611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:04:24.936956 11617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:24.937005 11619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:24.937047 11616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:04:24.937768 11611 server_base.cc:1034] running on GCE node
I20260808 04:04:24.938305 11611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:04:24.939164 11611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:04:24.940347 11611 hybrid_clock.cc:648] HybridClock initialized: now 1786161864940285 us; error 55 us; skew 500 ppm
I20260808 04:04:24.940621 11611 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:04:24 dist-test-slave-1l3l krb5kdc[11100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161864, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:04:24.947867 11611 init.cc:377] Logged in from keytab as kudu/127.5.83.3@KRBTEST.COM (short username kudu)
I20260808 04:04:24.948995 11611 fs_manager.cc:359] Metadata directory not provided
I20260808 04:04:24.949098 11611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:04:24.949385 11611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:04:24.952126 11611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/minicluster-data/ts-2/data/instance:
uuid: "73ffc4abb27d454499142bddf75a121f"
format_stamp: "Formatted at 2026-08-08 04:04:24 on dist-test-slave-1l3l"
server_key: "670d9ce7f22963f4524b04758c042810"
server_key_iv: "0aac8c94aa5b61fe42499c1c980be8d4"
server_key_version: "encryptionkey@0"
I20260808 04:04:24.952878 11611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/minicluster-data/ts-2/wal/instance:
uuid: "73ffc4abb27d454499142bddf75a121f"
format_stamp: "Formatted at 2026-08-08 04:04:24 on dist-test-slave-1l3l"
server_key: "670d9ce7f22963f4524b04758c042810"
server_key_iv: "0aac8c94aa5b61fe42499c1c980be8d4"
server_key_version: "encryptionkey@0"
I20260808 04:04:24.957448 11611 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260808 04:04:24.960595 11627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:24.961736 11611 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 04:04:24.961890 11611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/minicluster-data/ts-2/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/minicluster-data/ts-2/wal
uuid: "73ffc4abb27d454499142bddf75a121f"
format_stamp: "Formatted at 2026-08-08 04:04:24 on dist-test-slave-1l3l"
server_key: "670d9ce7f22963f4524b04758c042810"
server_key_iv: "0aac8c94aa5b61fe42499c1c980be8d4"
server_key_version: "encryptionkey@0"
I20260808 04:04:24.962085 11611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:04:24.975648 11611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:04:24.979462 11611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:04:24.979733 11611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:04:24.980598 11611 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:04:24.981863 11611 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:04:24.981978 11611 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:24.982064 11611 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:04:24.982120 11611 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:25.012719 11611 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.3:40569
I20260808 04:04:25.012774 11783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.3:40569 every 8 connection(s)
I20260808 04:04:25.013554 11611 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
I20260808 04:04:25.021427 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 11611
I20260808 04:04:25.021631 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/minicluster-data/ts-2/wal/instance
I20260808 04:04:25.022050 5452 external_mini_cluster.cc:1428] Setting key 4d27b6cdd80349de78612e5fa62e023a
Aug 08 04:04:25 dist-test-slave-1l3l krb5kdc[11100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161864, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.3@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:04:25.033551 11421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:25.015868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.3:52103 (local address 127.5.83.62:33649)
0808 04:04:25.016028 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:25.016033 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:25.017158 (+ 1125us) server_negotiation.cc:407] Connection header received
0808 04:04:25.018892 (+ 1734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:25.018899 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:25.018978 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:25.019065 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:25.020564 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:25.021474 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:25.023171 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:25.023769 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:25.023891 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:25.027313 (+ 3422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:25.027318 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:25.027321 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:25.027351 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:25.030621 (+ 3270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:25.031498 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:25.031521 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:25.031525 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:25.031649 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:25.032154 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:25.032160 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:25.032163 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:25.032485 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:25.032594 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:25.033160 (+ 566us) server_negotiation.cc:299] Negotiation successful
0808 04:04:25.033312 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 04:04:25.034389 11786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:25.016317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:33649 (local address 127.5.83.3:52103)
0808 04:04:25.017008 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:25.017046 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:25.018632 (+ 1586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:25.019307 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:25.019314 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:25.019846 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:04:25.020400 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:25.020417 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:25.021632 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:25.021637 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:25.023036 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:25.023050 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:25.023998 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:25.024006 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:25.024277 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:25.025099 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:04:25.025126 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:04:25.027121 (+ 1995us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:04:25.030783 (+ 3662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:25.030792 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:25.030806 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:25.031397 (+ 591us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:25.031786 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:25.031791 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:25.031795 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:25.032035 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:25.032598 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:04:25.032604 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:04:25.032877 (+ 273us) client_negotiation.cc:769] Sending connection context
0808 04:04:25.033094 (+ 217us) client_negotiation.cc:240] Negotiation successful
0808 04:04:25.033420 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":266,"threads_started":1}
I20260808 04:04:25.035634 11784 heartbeater.cc:347] Connected to a master server at 127.5.83.62:33649
I20260808 04:04:25.035951 11784 heartbeater.cc:464] Registering TS with master...
I20260808 04:04:25.036677 11784 heartbeater.cc:511] Master 127.5.83.62:33649 requested a full tablet report, sending...
I20260808 04:04:25.037988 11159 ts_manager.cc:194] Registered new tserver with Master: 73ffc4abb27d454499142bddf75a121f (127.5.83.3:40569)
I20260808 04:04:25.038781 11159 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.3@KRBTEST.COM'} at 127.5.83.3:52103
I20260808 04:04:25.048760 5452 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:04:25.064884 11421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:25.052945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40422 (local address 127.5.83.62:33649)
0808 04:04:25.053122 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:25.053127 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:25.053597 (+ 470us) server_negotiation.cc:407] Connection header received
0808 04:04:25.053724 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:25.053727 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:25.053794 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:25.053865 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:25.055207 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:25.055941 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:25.057070 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:25.057485 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:25.057568 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:25.059728 (+ 2160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:25.059736 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:25.059740 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:25.059782 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:25.062390 (+ 2608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:25.063032 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:25.063037 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:25.063041 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:25.063155 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:25.063490 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:25.063495 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:25.063498 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:25.063821 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:25.064037 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:25.064407 (+ 370us) server_negotiation.cc:299] Negotiation successful
0808 04:04:25.064605 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 04:04:25.068328 11159 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40422:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:04:25.075022 11159 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:04:25.091215 11419 heartbeater.cc:503] Master 127.5.83.62:33649 was elected leader, sending a full tablet report...
I20260808 04:04:25.107329 11803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:25.086962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44878 (local address 127.5.83.2:33045)
0808 04:04:25.099938 (+ 12976us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:25.099947 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:25.099975 (+ 28us) server_negotiation.cc:407] Connection header received
0808 04:04:25.100065 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:25.100073 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:25.100306 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:25.100506 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:04:25.101200 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:25.102801 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:25.105054 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:25.106272 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:25.106389 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:25.106588 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:25.106864 (+ 276us) server_negotiation.cc:299] Negotiation successful
0808 04:04:25.107029 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12792,"thread_start_us":12824,"threads_started":1}
I20260808 04:04:25.107389 11801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:25.086295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.2:33045 (local address 127.0.0.1:44878)
0808 04:04:25.098697 (+ 12402us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:25.098713 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:25.098805 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:25.100533 (+ 1728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:25.100540 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:25.100571 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:04:25.100992 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:25.101011 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:25.102957 (+ 1946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:25.102963 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:25.104836 (+ 1873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:25.104848 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:25.106466 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:25.106472 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:25.106655 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:25.106690 (+ 35us) client_negotiation.cc:769] Sending connection context
0808 04:04:25.106795 (+ 105us) client_negotiation.cc:240] Negotiation successful
0808 04:04:25.106972 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12292,"mutex_wait_us":121,"thread_start_us":11099,"threads_started":1}
I20260808 04:04:25.108229 11798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:25.086274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.3:40569 (local address 127.0.0.1:36848)
0808 04:04:25.098010 (+ 11736us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:25.098056 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:25.098211 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:25.100982 (+ 2771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:25.100986 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:25.101011 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:04:25.101438 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:25.101450 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:25.103170 (+ 1720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:25.103178 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:25.105568 (+ 2390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:25.105579 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:25.107363 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:25.107371 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:25.107556 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:25.107592 (+ 36us) client_negotiation.cc:769] Sending connection context
0808 04:04:25.107654 (+ 62us) client_negotiation.cc:240] Negotiation successful
0808 04:04:25.107796 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11508,"mutex_wait_us":20,"thread_start_us":11325,"threads_started":1}
I20260808 04:04:25.108338 11800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:25.086756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:40567 (local address 127.0.0.1:41724)
0808 04:04:25.099256 (+ 12500us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:25.099272 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:25.099366 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:25.100756 (+ 1390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:25.100759 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:25.100779 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:04:25.101221 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:25.101237 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:25.103039 (+ 1802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:25.103045 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:25.105565 (+ 2520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:25.105580 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:25.107598 (+ 2018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:25.107603 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:25.107773 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:25.107802 (+ 29us) client_negotiation.cc:769] Sending connection context
0808 04:04:25.107877 (+ 75us) client_negotiation.cc:240] Negotiation successful
0808 04:04:25.108007 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12385,"mutex_wait_us":144,"thread_start_us":12108,"threads_started":1}
I20260808 04:04:25.108635 11802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:25.086747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36848 (local address 127.5.83.3:40569)
0808 04:04:25.100454 (+ 13707us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:25.100462 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:25.100494 (+ 32us) server_negotiation.cc:407] Connection header received
0808 04:04:25.100585 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:25.100591 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:25.100796 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:25.100987 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:04:25.101649 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:25.102980 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:25.105752 (+ 2772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:25.107124 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:25.107879 (+ 755us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:25.108093 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:25.108168 (+ 75us) server_negotiation.cc:299] Negotiation successful
0808 04:04:25.108312 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13562,"thread_start_us":13389,"threads_started":1}
I20260808 04:04:25.109845 11804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:25.087525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41724 (local address 127.5.83.1:40567)
0808 04:04:25.099942 (+ 12417us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:25.099949 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:25.099974 (+ 25us) server_negotiation.cc:407] Connection header received
0808 04:04:25.100065 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:25.100073 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:25.100306 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:25.100499 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:04:25.101437 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:25.102933 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:25.105744 (+ 2811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:25.107126 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:25.107246 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:25.107460 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:25.108652 (+ 1192us) server_negotiation.cc:299] Negotiation successful
0808 04:04:25.109506 (+ 854us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12255,"thread_start_us":12065,"threads_started":1}
I20260808 04:04:25.111639 11511 tablet_service.cc:1467] Processing CreateTablet for tablet 3b1dc3ac841f4db49aa28bc1713d6979 (DEFAULT_TABLE table=test-table [id=612e6ef571c0420687202b5f8a4c0d16]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:04:25.112071 11690 tablet_service.cc:1467] Processing CreateTablet for tablet 3b1dc3ac841f4db49aa28bc1713d6979 (DEFAULT_TABLE table=test-table [id=612e6ef571c0420687202b5f8a4c0d16]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:04:25.113179 11511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b1dc3ac841f4db49aa28bc1713d6979. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:04:25.113333 11690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b1dc3ac841f4db49aa28bc1713d6979. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:04:25.113437 11331 tablet_service.cc:1467] Processing CreateTablet for tablet 3b1dc3ac841f4db49aa28bc1713d6979 (DEFAULT_TABLE table=test-table [id=612e6ef571c0420687202b5f8a4c0d16]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:04:25.114711 11331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b1dc3ac841f4db49aa28bc1713d6979. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:04:25.133255 11805 tablet_bootstrap.cc:492] T 3b1dc3ac841f4db49aa28bc1713d6979 P 73ffc4abb27d454499142bddf75a121f: Bootstrap starting.
I20260808 04:04:25.136409 11806 tablet_bootstrap.cc:492] T 3b1dc3ac841f4db49aa28bc1713d6979 P 70a2055fb6ae45289232a023174c2a78: Bootstrap starting.
I20260808 04:04:25.138314 11805 tablet_bootstrap.cc:654] T 3b1dc3ac841f4db49aa28bc1713d6979 P 73ffc4abb27d454499142bddf75a121f: Neither blocks nor log segments found. Creating new log.
I20260808 04:04:25.139138 11806 tablet_bootstrap.cc:654] T 3b1dc3ac841f4db49aa28bc1713d6979 P 70a2055fb6ae45289232a023174c2a78: Neither blocks nor log segments found. Creating new log.
I20260808 04:04:25.139814 11805 log.cc:824] T 3b1dc3ac841f4db49aa28bc1713d6979 P 73ffc4abb27d454499142bddf75a121f: Log is configured to *not* fsync() on all Append() calls
I20260808 04:04:25.140192 11806 log.cc:824] T 3b1dc3ac841f4db49aa28bc1713d6979 P 70a2055fb6ae45289232a023174c2a78: Log is configured to *not* fsync() on all Append() calls
I20260808 04:04:25.140625 11807 tablet_bootstrap.cc:492] T 3b1dc3ac841f4db49aa28bc1713d6979 P 671a3fb6cf3b489d9ae8d895846b8fd5: Bootstrap starting.
I20260808 04:04:25.144598 11807 tablet_bootstrap.cc:654] T 3b1dc3ac841f4db49aa28bc1713d6979 P 671a3fb6cf3b489d9ae8d895846b8fd5: Neither blocks nor log segments found. Creating new log.
I20260808 04:04:25.145713 11807 log.cc:824] T 3b1dc3ac841f4db49aa28bc1713d6979 P 671a3fb6cf3b489d9ae8d895846b8fd5: Log is configured to *not* fsync() on all Append() calls
I20260808 04:04:25.146858 11805 tablet_bootstrap.cc:492] T 3b1dc3ac841f4db49aa28bc1713d6979 P 73ffc4abb27d454499142bddf75a121f: No bootstrap required, opened a new log
I20260808 04:04:25.147161 11805 ts_tablet_manager.cc:1397] T 3b1dc3ac841f4db49aa28bc1713d6979 P 73ffc4abb27d454499142bddf75a121f: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.000s
I20260808 04:04:25.147439 11806 tablet_bootstrap.cc:492] T 3b1dc3ac841f4db49aa28bc1713d6979 P 70a2055fb6ae45289232a023174c2a78: No bootstrap required, opened a new log
I20260808 04:04:25.147758 11806 ts_tablet_manager.cc:1397] T 3b1dc3ac841f4db49aa28bc1713d6979 P 70a2055fb6ae45289232a023174c2a78: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.005s
I20260808 04:04:25.148876 11807 tablet_bootstrap.cc:492] T 3b1dc3ac841f4db49aa28bc1713d6979 P 671a3fb6cf3b489d9ae8d895846b8fd5: No bootstrap required, opened a new log
I20260808 04:04:25.149138 11807 ts_tablet_manager.cc:1397] T 3b1dc3ac841f4db49aa28bc1713d6979 P 671a3fb6cf3b489d9ae8d895846b8fd5: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.005s
I20260808 04:04:25.154526 11805 raft_consensus.cc:348] T 3b1dc3ac841f4db49aa28bc1713d6979 P 73ffc4abb27d454499142bddf75a121f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73ffc4abb27d454499142bddf75a121f" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 40569 } } peers { permanent_uuid: "70a2055fb6ae45289232a023174c2a78" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 33045 } } peers { permanent_uuid: "671a3fb6cf3b489d9ae8d895846b8fd5" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 40567 } }
I20260808 04:04:25.154894 11805 raft_consensus.cc:374] T 3b1dc3ac841f4db49aa28bc1713d6979 P 73ffc4abb27d454499142bddf75a121f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:04:25.154995 11805 raft_consensus.cc:729] T 3b1dc3ac841f4db49aa28bc1713d6979 P 73ffc4abb27d454499142bddf75a121f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73ffc4abb27d454499142bddf75a121f, State: Initialized, Role: FOLLOWER
I20260808 04:04:25.155612 11805 consensus_queue.cc:260] T 3b1dc3ac841f4db49aa28bc1713d6979 P 73ffc4abb27d454499142bddf75a121f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73ffc4abb27d454499142bddf75a121f" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 40569 } } peers { permanent_uuid: "70a2055fb6ae45289232a023174c2a78" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 33045 } } peers { permanent_uuid: "671a3fb6cf3b489d9ae8d895846b8fd5" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 40567 } }
I20260808 04:04:25.156688 11784 heartbeater.cc:503] Master 127.5.83.62:33649 was elected leader, sending a full tablet report...
I20260808 04:04:25.157155 11805 ts_tablet_manager.cc:1428] T 3b1dc3ac841f4db49aa28bc1713d6979 P 73ffc4abb27d454499142bddf75a121f: Time spent starting tablet: real 0.010s user 0.008s sys 0.004s
I20260808 04:04:25.158926 11807 raft_consensus.cc:348] T 3b1dc3ac841f4db49aa28bc1713d6979 P 671a3fb6cf3b489d9ae8d895846b8fd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73ffc4abb27d454499142bddf75a121f" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 40569 } } peers { permanent_uuid: "70a2055fb6ae45289232a023174c2a78" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 33045 } } peers { permanent_uuid: "671a3fb6cf3b489d9ae8d895846b8fd5" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 40567 } }
I20260808 04:04:25.159340 11807 raft_consensus.cc:374] T 3b1dc3ac841f4db49aa28bc1713d6979 P 671a3fb6cf3b489d9ae8d895846b8fd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:04:25.159452 11807 raft_consensus.cc:729] T 3b1dc3ac841f4db49aa28bc1713d6979 P 671a3fb6cf3b489d9ae8d895846b8fd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 671a3fb6cf3b489d9ae8d895846b8fd5, State: Initialized, Role: FOLLOWER
I20260808 04:04:25.159855 11806 raft_consensus.cc:348] T 3b1dc3ac841f4db49aa28bc1713d6979 P 70a2055fb6ae45289232a023174c2a78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73ffc4abb27d454499142bddf75a121f" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 40569 } } peers { permanent_uuid: "70a2055fb6ae45289232a023174c2a78" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 33045 } } peers { permanent_uuid: "671a3fb6cf3b489d9ae8d895846b8fd5" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 40567 } }
I20260808 04:04:25.160059 11807 consensus_queue.cc:260] T 3b1dc3ac841f4db49aa28bc1713d6979 P 671a3fb6cf3b489d9ae8d895846b8fd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73ffc4abb27d454499142bddf75a121f" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 40569 } } peers { permanent_uuid: "70a2055fb6ae45289232a023174c2a78" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 33045 } } peers { permanent_uuid: "671a3fb6cf3b489d9ae8d895846b8fd5" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 40567 } }
I20260808 04:04:25.160359 11806 raft_consensus.cc:374] T 3b1dc3ac841f4db49aa28bc1713d6979 P 70a2055fb6ae45289232a023174c2a78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:04:25.160470 11806 raft_consensus.cc:729] T 3b1dc3ac841f4db49aa28bc1713d6979 P 70a2055fb6ae45289232a023174c2a78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70a2055fb6ae45289232a023174c2a78, State: Initialized, Role: FOLLOWER
I20260808 04:04:25.161113 11806 consensus_queue.cc:260] T 3b1dc3ac841f4db49aa28bc1713d6979 P 70a2055fb6ae45289232a023174c2a78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73ffc4abb27d454499142bddf75a121f" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 40569 } } peers { permanent_uuid: "70a2055fb6ae45289232a023174c2a78" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 33045 } } peers { permanent_uuid: "671a3fb6cf3b489d9ae8d895846b8fd5" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 40567 } }
I20260808 04:04:25.162174 11606 heartbeater.cc:503] Master 127.5.83.62:33649 was elected leader, sending a full tablet report...
I20260808 04:04:25.162818 11806 ts_tablet_manager.cc:1428] T 3b1dc3ac841f4db49aa28bc1713d6979 P 70a2055fb6ae45289232a023174c2a78: Time spent starting tablet: real 0.015s user 0.010s sys 0.002s
I20260808 04:04:25.163213 11807 ts_tablet_manager.cc:1428] T 3b1dc3ac841f4db49aa28bc1713d6979 P 671a3fb6cf3b489d9ae8d895846b8fd5: Time spent starting tablet: real 0.014s user 0.011s sys 0.003s
W20260808 04:04:25.264952 11785 tablet.cc:2369] T 3b1dc3ac841f4db49aa28bc1713d6979 P 73ffc4abb27d454499142bddf75a121f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 04:04:25.291970 11607 tablet.cc:2369] T 3b1dc3ac841f4db49aa28bc1713d6979 P 70a2055fb6ae45289232a023174c2a78: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 04:04:25.313127 11420 tablet.cc:2369] T 3b1dc3ac841f4db49aa28bc1713d6979 P 671a3fb6cf3b489d9ae8d895846b8fd5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:04:25.452772 11817 raft_consensus.cc:482] T 3b1dc3ac841f4db49aa28bc1713d6979 P 70a2055fb6ae45289232a023174c2a78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:04:25.453048 11817 raft_consensus.cc:504] T 3b1dc3ac841f4db49aa28bc1713d6979 P 70a2055fb6ae45289232a023174c2a78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73ffc4abb27d454499142bddf75a121f" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 40569 } } peers { permanent_uuid: "70a2055fb6ae45289232a023174c2a78" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 33045 } } peers { permanent_uuid: "671a3fb6cf3b489d9ae8d895846b8fd5" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 40567 } }
I20260808 04:04:25.454753 11817 leader_election.cc:288] T 3b1dc3ac841f4db49aa28bc1713d6979 P 70a2055fb6ae45289232a023174c2a78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73ffc4abb27d454499142bddf75a121f (127.5.83.3:40569), 671a3fb6cf3b489d9ae8d895846b8fd5 (127.5.83.1:40567)
I20260808 04:04:25.461822 11804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:25.455379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.2:41483 (local address 127.5.83.1:40567)
0808 04:04:25.455654 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:25.455660 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:25.456339 (+ 679us) server_negotiation.cc:407] Connection header received
0808 04:04:25.456539 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:25.456543 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:25.456604 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:25.456679 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:04:25.457282 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:25.458081 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:25.460140 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:25.460892 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:25.461025 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:25.461127 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:25.461468 (+ 341us) server_negotiation.cc:299] Negotiation successful
0808 04:04:25.461590 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
I20260808 04:04:25.461843 11802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:25.455268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.2:51217 (local address 127.5.83.3:40569)
0808 04:04:25.455518 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:25.455524 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:25.456416 (+ 892us) server_negotiation.cc:407] Connection header received
0808 04:04:25.456594 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:25.456600 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:25.456659 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:25.456732 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:04:25.457354 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:25.458085 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:25.460110 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:25.460845 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:25.460930 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:25.461006 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:25.461478 (+ 472us) server_negotiation.cc:299] Negotiation successful
0808 04:04:25.461608 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260808 04:04:25.462024 11819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:25.455191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.3:40569 (local address 127.5.83.2:51217)
0808 04:04:25.456312 (+ 1121us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:25.456330 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:25.456465 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:25.456755 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:25.456759 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:25.456782 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:04:25.457208 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:25.457217 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:25.458214 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:25.458218 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:25.459931 (+ 1713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:25.459946 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:25.461171 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:25.461175 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:25.461326 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:25.461347 (+ 21us) client_negotiation.cc:769] Sending connection context
0808 04:04:25.461423 (+ 76us) client_negotiation.cc:240] Negotiation successful
0808 04:04:25.461554 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1013,"mutex_wait_us":70,"thread_start_us":749,"threads_started":1}
I20260808 04:04:25.462292 11818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:25.455093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:40567 (local address 127.5.83.2:41483)
0808 04:04:25.456228 (+ 1135us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:25.456249 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:25.456385 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:25.456714 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:25.456719 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:25.456743 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:04:25.457125 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:25.457135 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:25.458215 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:25.458218 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:25.459994 (+ 1776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:25.460009 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:25.461142 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:25.461148 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:25.461317 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:25.461342 (+ 25us) client_negotiation.cc:769] Sending connection context
0808 04:04:25.461425 (+ 83us) client_negotiation.cc:240] Negotiation successful
0808 04:04:25.461560 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1020,"mutex_wait_us":65,"thread_start_us":545,"threads_started":1}
I20260808 04:04:25.463145 11718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b1dc3ac841f4db49aa28bc1713d6979" candidate_uuid: "70a2055fb6ae45289232a023174c2a78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73ffc4abb27d454499142bddf75a121f" is_pre_election: true
I20260808 04:04:25.463152 11359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b1dc3ac841f4db49aa28bc1713d6979" candidate_uuid: "70a2055fb6ae45289232a023174c2a78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "671a3fb6cf3b489d9ae8d895846b8fd5" is_pre_election: true
I20260808 04:04:25.463689 11718 raft_consensus.cc:2454] T 3b1dc3ac841f4db49aa28bc1713d6979 P 73ffc4abb27d454499142bddf75a121f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70a2055fb6ae45289232a023174c2a78 in term 0.
I20260808 04:04:25.463688 11359 raft_consensus.cc:2454] T 3b1dc3ac841f4db49aa28bc1713d6979 P 671a3fb6cf3b489d9ae8d895846b8fd5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70a2055fb6ae45289232a023174c2a78 in term 0.
I20260808 04:04:25.464344 11451 leader_election.cc:302] T 3b1dc3ac841f4db49aa28bc1713d6979 P 70a2055fb6ae45289232a023174c2a78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70a2055fb6ae45289232a023174c2a78, 73ffc4abb27d454499142bddf75a121f; no voters:
I20260808 04:04:25.464700 11817 raft_consensus.cc:2781] T 3b1dc3ac841f4db49aa28bc1713d6979 P 70a2055fb6ae45289232a023174c2a78 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:04:25.464833 11817 raft_consensus.cc:482] T 3b1dc3ac841f4db49aa28bc1713d6979 P 70a2055fb6ae45289232a023174c2a78 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:04:25.464932 11817 raft_consensus.cc:3037] T 3b1dc3ac841f4db49aa28bc1713d6979 P 70a2055fb6ae45289232a023174c2a78 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:04:25.467424 11817 raft_consensus.cc:504] T 3b1dc3ac841f4db49aa28bc1713d6979 P 70a2055fb6ae45289232a023174c2a78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73ffc4abb27d454499142bddf75a121f" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 40569 } } peers { permanent_uuid: "70a2055fb6ae45289232a023174c2a78" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 33045 } } peers { permanent_uuid: "671a3fb6cf3b489d9ae8d895846b8fd5" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 40567 } }
I20260808 04:04:25.467993 11817 leader_election.cc:288] T 3b1dc3ac841f4db49aa28bc1713d6979 P 70a2055fb6ae45289232a023174c2a78 [CANDIDATE]: Term 1 election: Requested vote from peers 73ffc4abb27d454499142bddf75a121f (127.5.83.3:40569), 671a3fb6cf3b489d9ae8d895846b8fd5 (127.5.83.1:40567)
I20260808 04:04:25.468374 11718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b1dc3ac841f4db49aa28bc1713d6979" candidate_uuid: "70a2055fb6ae45289232a023174c2a78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73ffc4abb27d454499142bddf75a121f"
I20260808 04:04:25.468400 11359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b1dc3ac841f4db49aa28bc1713d6979" candidate_uuid: "70a2055fb6ae45289232a023174c2a78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "671a3fb6cf3b489d9ae8d895846b8fd5"
I20260808 04:04:25.468547 11718 raft_consensus.cc:3037] T 3b1dc3ac841f4db49aa28bc1713d6979 P 73ffc4abb27d454499142bddf75a121f [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:04:25.468555 11359 raft_consensus.cc:3037] T 3b1dc3ac841f4db49aa28bc1713d6979 P 671a3fb6cf3b489d9ae8d895846b8fd5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:04:25.470777 11718 raft_consensus.cc:2454] T 3b1dc3ac841f4db49aa28bc1713d6979 P 73ffc4abb27d454499142bddf75a121f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70a2055fb6ae45289232a023174c2a78 in term 1.
I20260808 04:04:25.470804 11359 raft_consensus.cc:2454] T 3b1dc3ac841f4db49aa28bc1713d6979 P 671a3fb6cf3b489d9ae8d895846b8fd5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70a2055fb6ae45289232a023174c2a78 in term 1.
I20260808 04:04:25.471146 11456 leader_election.cc:302] T 3b1dc3ac841f4db49aa28bc1713d6979 P 70a2055fb6ae45289232a023174c2a78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 671a3fb6cf3b489d9ae8d895846b8fd5, 70a2055fb6ae45289232a023174c2a78, 73ffc4abb27d454499142bddf75a121f; no voters:
I20260808 04:04:25.471452 11817 raft_consensus.cc:2781] T 3b1dc3ac841f4db49aa28bc1713d6979 P 70a2055fb6ae45289232a023174c2a78 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:04:25.471885 11817 raft_consensus.cc:686] T 3b1dc3ac841f4db49aa28bc1713d6979 P 70a2055fb6ae45289232a023174c2a78 [term 1 LEADER]: Becoming Leader. State: Replica: 70a2055fb6ae45289232a023174c2a78, State: Running, Role: LEADER
I20260808 04:04:25.472343 11817 consensus_queue.cc:237] T 3b1dc3ac841f4db49aa28bc1713d6979 P 70a2055fb6ae45289232a023174c2a78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73ffc4abb27d454499142bddf75a121f" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 40569 } } peers { permanent_uuid: "70a2055fb6ae45289232a023174c2a78" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 33045 } } peers { permanent_uuid: "671a3fb6cf3b489d9ae8d895846b8fd5" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 40567 } }
I20260808 04:04:25.476438 11159 catalog_manager.cc:5495] T 3b1dc3ac841f4db49aa28bc1713d6979 P 70a2055fb6ae45289232a023174c2a78 reported cstate change: term changed from 0 to 1, leader changed from <none> to 70a2055fb6ae45289232a023174c2a78 (127.5.83.2). New cstate: current_term: 1 leader_uuid: "70a2055fb6ae45289232a023174c2a78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73ffc4abb27d454499142bddf75a121f" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 40569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "70a2055fb6ae45289232a023174c2a78" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 33045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "671a3fb6cf3b489d9ae8d895846b8fd5" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 40567 } health_report { overall_health: UNKNOWN } } }
I20260808 04:04:25.566071 11803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:25.561266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44888 (local address 127.5.83.2:33045)
0808 04:04:25.561442 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:25.561447 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:25.561524 (+ 77us) server_negotiation.cc:407] Connection header received
0808 04:04:25.561685 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:25.561689 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:25.561768 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:25.561882 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:04:25.562172 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:25.562880 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:25.564169 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:25.564561 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:25.564650 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:25.565077 (+ 427us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:04:25.565586 (+ 509us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:04:25.565693 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:25.565776 (+ 83us) server_negotiation.cc:299] Negotiation successful
0808 04:04:25.565838 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 04:04:25.573956 11359 raft_consensus.cc:1261] T 3b1dc3ac841f4db49aa28bc1713d6979 P 671a3fb6cf3b489d9ae8d895846b8fd5 [term 1 FOLLOWER]: Refusing update from remote peer 70a2055fb6ae45289232a023174c2a78: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:04:25.573956 11718 raft_consensus.cc:1261] T 3b1dc3ac841f4db49aa28bc1713d6979 P 73ffc4abb27d454499142bddf75a121f [term 1 FOLLOWER]: Refusing update from remote peer 70a2055fb6ae45289232a023174c2a78: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:04:25.575032 11820 consensus_queue.cc:1035] T 3b1dc3ac841f4db49aa28bc1713d6979 P 70a2055fb6ae45289232a023174c2a78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "671a3fb6cf3b489d9ae8d895846b8fd5" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 40567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:04:25.575379 11817 consensus_queue.cc:1035] T 3b1dc3ac841f4db49aa28bc1713d6979 P 70a2055fb6ae45289232a023174c2a78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73ffc4abb27d454499142bddf75a121f" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 40569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:04:25.600018 11158 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:04:25.600301 11158 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:04:25.601327 11825 mvcc.cc:205] Tried to move back new op lower bound from 7316119001380134912 to 7316119000977940480. Current Snapshot: MvccSnapshot[applied={T|T < 7316119001380134912}]
I20260808 04:04:25.604550 11158 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b81776e15e0c4e3cb5c378df1fa84c24: Sending DeleteTablet for 3 replicas of tablet 3b1dc3ac841f4db49aa28bc1713d6979
I20260808 04:04:25.605721 11690 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b1dc3ac841f4db49aa28bc1713d6979 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:04:25 UTC) from {username='kudu', principal='kudu/127.5.83.62@KRBTEST.COM'} at 127.0.0.1:36848
I20260808 04:04:25.605726 11329 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b1dc3ac841f4db49aa28bc1713d6979 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:04:25 UTC) from {username='kudu', principal='kudu/127.5.83.62@KRBTEST.COM'} at 127.0.0.1:41724
I20260808 04:04:25.606160 11511 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b1dc3ac841f4db49aa28bc1713d6979 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:04:25 UTC) from {username='kudu', principal='kudu/127.5.83.62@KRBTEST.COM'} at 127.0.0.1:44878
I20260808 04:04:25.606632 11331 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b1dc3ac841f4db49aa28bc1713d6979 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:04:25 UTC) from {username='kudu', principal='kudu/127.5.83.62@KRBTEST.COM'} at 127.0.0.1:41724
I20260808 04:04:25.608889 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 11246
I20260808 04:04:25.616608 11836 tablet_replica.cc:330] T 3b1dc3ac841f4db49aa28bc1713d6979 P 70a2055fb6ae45289232a023174c2a78: stopping tablet replica
I20260808 04:04:25.617735 11836 raft_consensus.cc:2229] T 3b1dc3ac841f4db49aa28bc1713d6979 P 70a2055fb6ae45289232a023174c2a78 [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:04:25.618974 11836 raft_consensus.cc:2258] T 3b1dc3ac841f4db49aa28bc1713d6979 P 70a2055fb6ae45289232a023174c2a78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:04:25.621271 11835 tablet_replica.cc:330] T 3b1dc3ac841f4db49aa28bc1713d6979 P 671a3fb6cf3b489d9ae8d895846b8fd5: stopping tablet replica
I20260808 04:04:25.621716 11835 raft_consensus.cc:2229] T 3b1dc3ac841f4db49aa28bc1713d6979 P 671a3fb6cf3b489d9ae8d895846b8fd5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:04:25.622221 11835 raft_consensus.cc:2258] T 3b1dc3ac841f4db49aa28bc1713d6979 P 671a3fb6cf3b489d9ae8d895846b8fd5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:04:25.623174 11836 ts_tablet_manager.cc:1905] T 3b1dc3ac841f4db49aa28bc1713d6979 P 70a2055fb6ae45289232a023174c2a78: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:04:25.623450 11834 tablet_replica.cc:330] T 3b1dc3ac841f4db49aa28bc1713d6979 P 73ffc4abb27d454499142bddf75a121f: stopping tablet replica
I20260808 04:04:25.623903 11834 raft_consensus.cc:2229] T 3b1dc3ac841f4db49aa28bc1713d6979 P 73ffc4abb27d454499142bddf75a121f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:04:25.624142 11835 ts_tablet_manager.cc:1905] T 3b1dc3ac841f4db49aa28bc1713d6979 P 671a3fb6cf3b489d9ae8d895846b8fd5: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 08 04:04:25 dist-test-slave-1l3l krb5kdc[11100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.1@KRBTEST.COM
I20260808 04:04:25.626950 11834 raft_consensus.cc:2258] T 3b1dc3ac841f4db49aa28bc1713d6979 P 73ffc4abb27d454499142bddf75a121f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:04:25.631166 11835 ts_tablet_manager.cc:1918] T 3b1dc3ac841f4db49aa28bc1713d6979 P 671a3fb6cf3b489d9ae8d895846b8fd5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:04:25.631345 11835 log.cc:1196] T 3b1dc3ac841f4db49aa28bc1713d6979 P 671a3fb6cf3b489d9ae8d895846b8fd5: Deleting WAL directory at /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/minicluster-data/ts-0/wal/wals/3b1dc3ac841f4db49aa28bc1713d6979
I20260808 04:04:25.631916 11835 ts_tablet_manager.cc:1939] T 3b1dc3ac841f4db49aa28bc1713d6979 P 671a3fb6cf3b489d9ae8d895846b8fd5: Deleting consensus metadata
I20260808 04:04:25.633441 11834 ts_tablet_manager.cc:1905] T 3b1dc3ac841f4db49aa28bc1713d6979 P 73ffc4abb27d454499142bddf75a121f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:04:25.633733 11143 catalog_manager.cc:4841] TS 671a3fb6cf3b489d9ae8d895846b8fd5 (127.5.83.1:40567): tablet 3b1dc3ac841f4db49aa28bc1713d6979 (table test-table [id=612e6ef571c0420687202b5f8a4c0d16]) successfully deleted
W20260808 04:04:25.634927 11143 catalog_manager.cc:4808] TS 671a3fb6cf3b489d9ae8d895846b8fd5 (127.5.83.1:40567): delete failed for tablet 3b1dc3ac841f4db49aa28bc1713d6979 because the tablet was not found. No further retry: Not found: Tablet not found: 3b1dc3ac841f4db49aa28bc1713d6979
I20260808 04:04:25.635855 11804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:25.610647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41732 (local address 127.5.83.1:40567)
0808 04:04:25.610839 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:25.610846 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:25.614542 (+ 3696us) server_negotiation.cc:407] Connection header received
0808 04:04:25.614720 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:25.614726 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:25.614831 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:25.614934 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:25.617921 (+ 2987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:25.619123 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:25.621222 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:25.621949 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:25.622053 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:25.627917 (+ 5864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:25.627933 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:25.627951 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:25.627999 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:25.632242 (+ 4243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:25.633130 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:25.633139 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:25.633143 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:25.633326 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:25.633812 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:25.633817 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:25.633821 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:25.634265 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:25.634593 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:25.635198 (+ 605us) server_negotiation.cc:299] Negotiation successful
0808 04:04:25.635453 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 04:04:25.636754 11836 ts_tablet_manager.cc:1918] T 3b1dc3ac841f4db49aa28bc1713d6979 P 70a2055fb6ae45289232a023174c2a78: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:04:25.636955 11836 log.cc:1196] T 3b1dc3ac841f4db49aa28bc1713d6979 P 70a2055fb6ae45289232a023174c2a78: Deleting WAL directory at /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/minicluster-data/ts-1/wal/wals/3b1dc3ac841f4db49aa28bc1713d6979
I20260808 04:04:25.637557 11836 ts_tablet_manager.cc:1939] T 3b1dc3ac841f4db49aa28bc1713d6979 P 70a2055fb6ae45289232a023174c2a78: Deleting consensus metadata
I20260808 04:04:25.637615 11414 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:04:25.639675 11140 catalog_manager.cc:4841] TS 70a2055fb6ae45289232a023174c2a78 (127.5.83.2:33045): tablet 3b1dc3ac841f4db49aa28bc1713d6979 (table test-table [id=612e6ef571c0420687202b5f8a4c0d16]) successfully deleted
I20260808 04:04:25.644480 11834 ts_tablet_manager.cc:1918] T 3b1dc3ac841f4db49aa28bc1713d6979 P 73ffc4abb27d454499142bddf75a121f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:04:25.644632 11834 log.cc:1196] T 3b1dc3ac841f4db49aa28bc1713d6979 P 73ffc4abb27d454499142bddf75a121f: Deleting WAL directory at /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161826605655-5452-0/minicluster-data/ts-2/wal/wals/3b1dc3ac841f4db49aa28bc1713d6979
I20260808 04:04:25.645042 11834 ts_tablet_manager.cc:1939] T 3b1dc3ac841f4db49aa28bc1713d6979 P 73ffc4abb27d454499142bddf75a121f: Deleting consensus metadata
I20260808 04:04:25.646392 11141 catalog_manager.cc:4841] TS 73ffc4abb27d454499142bddf75a121f (127.5.83.3:40569): tablet 3b1dc3ac841f4db49aa28bc1713d6979 (table test-table [id=612e6ef571c0420687202b5f8a4c0d16]) successfully deleted
I20260808 04:04:25.740921 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 11246
I20260808 04:04:25.765551 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 11427
Aug 08 04:04:25 dist-test-slave-1l3l krb5kdc[11100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.2@KRBTEST.COM
I20260808 04:04:25.782346 11803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:25.766405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44900 (local address 127.5.83.2:33045)
0808 04:04:25.766608 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:25.766614 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:25.766636 (+ 22us) server_negotiation.cc:407] Connection header received
0808 04:04:25.766733 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:25.766739 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:25.766836 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:25.766949 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:25.768166 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:25.769408 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:25.770685 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:25.771393 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:25.771549 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:25.775578 (+ 4029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:25.775595 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:25.775615 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:25.775658 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:25.779452 (+ 3794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:25.780150 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:25.780159 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:25.780163 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:25.780326 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:25.780660 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:25.780665 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:25.780668 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:25.781134 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:25.781510 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:25.781721 (+ 211us) server_negotiation.cc:299] Negotiation successful
0808 04:04:25.781939 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260808 04:04:25.783124 11597 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:04:25.892750 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 11427
I20260808 04:04:25.915992 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 11611
Aug 08 04:04:25 dist-test-slave-1l3l krb5kdc[11100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.3@KRBTEST.COM
I20260808 04:04:25.930236 11802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:25.916742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36856 (local address 127.5.83.3:40569)
0808 04:04:25.916924 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:25.916930 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:25.916951 (+ 21us) server_negotiation.cc:407] Connection header received
0808 04:04:25.917147 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:25.917151 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:25.917228 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:25.917314 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:25.918581 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:25.919283 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:25.920595 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:25.921003 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:25.921116 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:25.925193 (+ 4077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:25.925207 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:25.925228 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:25.925259 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:25.927691 (+ 2432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:25.928315 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:25.928323 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:25.928325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:25.928463 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:25.928813 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:25.928817 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:25.928821 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:25.929147 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:25.929436 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:25.929718 (+ 282us) server_negotiation.cc:299] Negotiation successful
0808 04:04:25.929926 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 04:04:25.930837 11779 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:04:26.016593 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 11611
I20260808 04:04:26.037675 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 11120
I20260808 04:04:26.038812 11203 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:04:26.146063 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 11120
2026-08-08T04:04:26Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4406 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-08T04:04:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:04:26Z Disabled control of system clock
I20260808 04:04:26.215377 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161826605655-5452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161826605655-5452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161826605655-5452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:36133
--webserver_interface=127.5.83.62
--webserver_port=0
--builtin_ntp_servers=127.5.83.20:35603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.62:36133
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260808 04:04:26.545516 11851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:04:26.545802 11851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:04:26.545929 11851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:04:26.553092 11851 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:04:26.553192 11851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:04:26.553254 11851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:04:26.553339 11851 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:04:26.553395 11851 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:04:26.562304 11851 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:35603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161826605655-5452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161826605655-5452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.62:36133
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:36133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161826605655-5452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:04:26.563504 11851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:04:26.565209 11851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:04:26.576061 11863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:26.576155 11858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:26.576134 11860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:04:26.576887 11851 server_base.cc:1034] running on GCE node
I20260808 04:04:26.577605 11851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:04:26.578855 11851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:04:26.580054 11851 hybrid_clock.cc:648] HybridClock initialized: now 1786161866580018 us; error 25 us; skew 500 ppm
I20260808 04:04:26.580359 11851 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 04:04:26.582417 11851 webserver.cc:460] Webserver started at http://127.5.83.62:45305/ using document root <none> and password file <none>
I20260808 04:04:26.583030 11851 fs_manager.cc:359] Metadata directory not provided
I20260808 04:04:26.583132 11851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:04:26.583437 11851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:04:26.586153 11851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161826605655-5452-0/minicluster-data/master-0/data/instance:
uuid: "7da96244aa394e2b942371d4dce3a145"
format_stamp: "Formatted at 2026-08-08 04:04:26 on dist-test-slave-1l3l"
server_key: "f3fb8e40e31283e15f33ba98e8092bc7"
server_key_iv: "f283cdd062c3be06c06ec45a2f2d9416"
server_key_version: "encryptionkey@0"
I20260808 04:04:26.586963 11851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161826605655-5452-0/minicluster-data/master-0/wal/instance:
uuid: "7da96244aa394e2b942371d4dce3a145"
format_stamp: "Formatted at 2026-08-08 04:04:26 on dist-test-slave-1l3l"
server_key: "f3fb8e40e31283e15f33ba98e8092bc7"
server_key_iv: "f283cdd062c3be06c06ec45a2f2d9416"
server_key_version: "encryptionkey@0"
I20260808 04:04:26.591624 11851 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260808 04:04:26.594731 11874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:26.595937 11851 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.003s
I20260808 04:04:26.596088 11851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161826605655-5452-0/minicluster-data/master-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161826605655-5452-0/minicluster-data/master-0/wal
uuid: "7da96244aa394e2b942371d4dce3a145"
format_stamp: "Formatted at 2026-08-08 04:04:26 on dist-test-slave-1l3l"
server_key: "f3fb8e40e31283e15f33ba98e8092bc7"
server_key_iv: "f283cdd062c3be06c06ec45a2f2d9416"
server_key_version: "encryptionkey@0"
I20260808 04:04:26.596278 11851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161826605655-5452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161826605655-5452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161826605655-5452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:04:26.601467 11851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:04:26.602144 11851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:04:26.602418 11851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:04:26.618521 11851 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.62:36133
I20260808 04:04:26.618558 11948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.62:36133 every 8 connection(s)
I20260808 04:04:26.619769 11851 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
I20260808 04:04:26.620918 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 11851
I20260808 04:04:26.621099 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161826605655-5452-0/minicluster-data/master-0/wal/instance
I20260808 04:04:26.621737 5452 external_mini_cluster.cc:1428] Setting key d9d1a46ac938a9cb751990b2c22301ed
I20260808 04:04:26.623778 11949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:04:26.630702 11952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:26.623644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46790 (local address 127.5.83.62:36133)
0808 04:04:26.624723 (+ 1079us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:26.624737 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:26.624786 (+ 49us) server_negotiation.cc:407] Connection header received
0808 04:04:26.626723 (+ 1937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:26.626750 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:26.627275 (+ 525us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:26.627697 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:26.628732 (+ 1035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:26.628741 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:26.628758 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:04:26.628774 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:26.628869 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:04:26.628906 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:26.629138 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:26.629269 (+ 131us) server_negotiation.cc:299] Negotiation successful
0808 04:04:26.629526 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":345,"threads_started":1}
I20260808 04:04:26.636348 11949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7da96244aa394e2b942371d4dce3a145: Bootstrap starting.
I20260808 04:04:26.639169 11949 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7da96244aa394e2b942371d4dce3a145: Neither blocks nor log segments found. Creating new log.
I20260808 04:04:26.640183 11949 log.cc:824] T 00000000000000000000000000000000 P 7da96244aa394e2b942371d4dce3a145: Log is configured to *not* fsync() on all Append() calls
I20260808 04:04:26.642769 11949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7da96244aa394e2b942371d4dce3a145: No bootstrap required, opened a new log
I20260808 04:04:26.649415 11949 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7da96244aa394e2b942371d4dce3a145 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7da96244aa394e2b942371d4dce3a145" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 36133 } }
I20260808 04:04:26.649721 11949 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7da96244aa394e2b942371d4dce3a145 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:04:26.649806 11949 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7da96244aa394e2b942371d4dce3a145 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7da96244aa394e2b942371d4dce3a145, State: Initialized, Role: FOLLOWER
I20260808 04:04:26.650324 11949 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7da96244aa394e2b942371d4dce3a145 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7da96244aa394e2b942371d4dce3a145" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 36133 } }
I20260808 04:04:26.650539 11949 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7da96244aa394e2b942371d4dce3a145 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:04:26.650658 11949 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7da96244aa394e2b942371d4dce3a145 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:04:26.650820 11949 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7da96244aa394e2b942371d4dce3a145 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:04:26.652858 11949 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7da96244aa394e2b942371d4dce3a145 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7da96244aa394e2b942371d4dce3a145" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 36133 } }
I20260808 04:04:26.653286 11949 leader_election.cc:302] T 00000000000000000000000000000000 P 7da96244aa394e2b942371d4dce3a145 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7da96244aa394e2b942371d4dce3a145; no voters:
I20260808 04:04:26.653789 11949 leader_election.cc:288] T 00000000000000000000000000000000 P 7da96244aa394e2b942371d4dce3a145 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:04:26.653894 11956 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7da96244aa394e2b942371d4dce3a145 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:04:26.654569 11956 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7da96244aa394e2b942371d4dce3a145 [term 1 LEADER]: Becoming Leader. State: Replica: 7da96244aa394e2b942371d4dce3a145, State: Running, Role: LEADER
I20260808 04:04:26.655078 11956 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7da96244aa394e2b942371d4dce3a145 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7da96244aa394e2b942371d4dce3a145" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 36133 } }
I20260808 04:04:26.655628 11949 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7da96244aa394e2b942371d4dce3a145 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:04:26.657732 11958 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7da96244aa394e2b942371d4dce3a145 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7da96244aa394e2b942371d4dce3a145. Latest consensus state: current_term: 1 leader_uuid: "7da96244aa394e2b942371d4dce3a145" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7da96244aa394e2b942371d4dce3a145" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 36133 } } }
I20260808 04:04:26.658155 11958 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7da96244aa394e2b942371d4dce3a145 [sys.catalog]: This master's current role is: LEADER
I20260808 04:04:26.657711 11957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7da96244aa394e2b942371d4dce3a145 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7da96244aa394e2b942371d4dce3a145" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7da96244aa394e2b942371d4dce3a145" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 36133 } } }
I20260808 04:04:26.658541 11957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7da96244aa394e2b942371d4dce3a145 [sys.catalog]: This master's current role is: LEADER
I20260808 04:04:26.659874 11963 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:04:26.664239 11963 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:04:26.672356 11963 catalog_manager.cc:1347] Generated new cluster ID: 876cd4bc3cf844e499d6763be5f888c4
I20260808 04:04:26.672461 11963 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:04:26.707264 11963 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:04:26.707769 11963 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:04:26.728150 11963 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7da96244aa394e2b942371d4dce3a145: Generated new TSK 0
I20260808 04:04:26.729142 11963 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:04:26.743292 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161826605655-5452-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161826605655-5452-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161826605655-5452-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.1:0
--local_ip_for_outbound_sockets=127.5.83.1
--webserver_interface=127.5.83.1
--webserver_port=0
--tserver_master_addrs=127.5.83.62:36133
--builtin_ntp_servers=127.5.83.20:35603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260808 04:04:27.060412 11982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:04:27.060664 11982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:04:27.060753 11982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:04:27.067672 11982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:04:27.067781 11982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:04:27.067894 11982 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.1
I20260808 04:04:27.076557 11982 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:35603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161826605655-5452-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161826605655-5452-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.83.1
--webserver_port=0
--tserver_master_addrs=127.5.83.62:36133
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.1
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161826605655-5452-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:04:27.077706 11982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:04:27.079383 11982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:04:27.091614 11987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:27.091634 11990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:27.091759 11988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:04:27.092694 11982 server_base.cc:1034] running on GCE node
I20260808 04:04:27.093194 11982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:04:27.093989 11982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:04:27.095191 11982 hybrid_clock.cc:648] HybridClock initialized: now 1786161867095142 us; error 35 us; skew 500 ppm
I20260808 04:04:27.095521 11982 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 04:04:27.097611 11982 webserver.cc:460] Webserver started at http://127.5.83.1:45897/ using document root <none> and password file <none>
I20260808 04:04:27.098258 11982 fs_manager.cc:359] Metadata directory not provided
I20260808 04:04:27.098408 11982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:04:27.098716 11982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:04:27.101595 11982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161826605655-5452-0/minicluster-data/ts-0/data/instance:
uuid: "dd3c0452629247f8b6279bed425b57b1"
format_stamp: "Formatted at 2026-08-08 04:04:27 on dist-test-slave-1l3l"
server_key: "a3f9fe3dc3a999203253957c97bb1160"
server_key_iv: "76d2850d052f053d3e452b48fcc5b266"
server_key_version: "encryptionkey@0"
I20260808 04:04:27.102418 11982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161826605655-5452-0/minicluster-data/ts-0/wal/instance:
uuid: "dd3c0452629247f8b6279bed425b57b1"
format_stamp: "Formatted at 2026-08-08 04:04:27 on dist-test-slave-1l3l"
server_key: "a3f9fe3dc3a999203253957c97bb1160"
server_key_iv: "76d2850d052f053d3e452b48fcc5b266"
server_key_version: "encryptionkey@0"
I20260808 04:04:27.107124 11982 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260808 04:04:27.110440 11999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:27.111616 11982 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:04:27.111797 11982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161826605655-5452-0/minicluster-data/ts-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161826605655-5452-0/minicluster-data/ts-0/wal
uuid: "dd3c0452629247f8b6279bed425b57b1"
format_stamp: "Formatted at 2026-08-08 04:04:27 on dist-test-slave-1l3l"
server_key: "a3f9fe3dc3a999203253957c97bb1160"
server_key_iv: "76d2850d052f053d3e452b48fcc5b266"
server_key_version: "encryptionkey@0"
I20260808 04:04:27.112031 11982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161826605655-5452-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161826605655-5452-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161826605655-5452-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:04:27.117539 11982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:04:27.118224 11982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:04:27.118462 11982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:04:27.119268 11982 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:04:27.120654 11982 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:04:27.120744 11982 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:27.120823 11982 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:04:27.120877 11982 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:27.151553 11982 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.1:46841
I20260808 04:04:27.151623 12151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.1:46841 every 8 connection(s)
I20260808 04:04:27.152863 11982 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
I20260808 04:04:27.160652 12154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:27.155234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.1:54471 (local address 127.5.83.62:36133)
0808 04:04:27.155684 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:27.155689 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:27.156796 (+ 1107us) server_negotiation.cc:407] Connection header received
0808 04:04:27.158428 (+ 1632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:27.158433 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:27.158495 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:27.158593 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:27.159887 (+ 1294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:27.159893 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:27.159896 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:04:27.159912 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:27.159974 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:04:27.159986 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:27.160086 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:27.160272 (+ 186us) server_negotiation.cc:299] Negotiation successful
0808 04:04:27.160359 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":249,"threads_started":1}
I20260808 04:04:27.161504 12155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:27.155761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:36133 (local address 127.5.83.1:54471)
0808 04:04:27.156649 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:27.156690 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:27.158169 (+ 1479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:27.158823 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:27.158830 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:27.158920 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:04:27.159575 (+ 655us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:04:27.159578 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:04:27.159613 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:04:27.159693 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:04:27.160073 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:04:27.160082 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:04:27.160090 (+ 8us) client_negotiation.cc:769] Sending connection context
0808 04:04:27.160360 (+ 270us) client_negotiation.cc:240] Negotiation successful
0808 04:04:27.160612 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":294,"threads_started":1}
I20260808 04:04:27.162993 12152 heartbeater.cc:347] Connected to a master server at 127.5.83.62:36133
I20260808 04:04:27.163164 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 11982
I20260808 04:04:27.163257 12152 heartbeater.cc:464] Registering TS with master...
I20260808 04:04:27.163342 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161826605655-5452-0/minicluster-data/ts-0/wal/instance
I20260808 04:04:27.163854 12152 heartbeater.cc:511] Master 127.5.83.62:36133 requested a full tablet report, sending...
I20260808 04:04:27.164037 5452 external_mini_cluster.cc:1428] Setting key 89d3d417e983b30a1879bf56bd913b4a
I20260808 04:04:27.165892 11902 ts_manager.cc:194] Registered new tserver with Master: dd3c0452629247f8b6279bed425b57b1 (127.5.83.1:46841)
I20260808 04:04:27.167002 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161826605655-5452-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161826605655-5452-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161826605655-5452-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.2:0
--local_ip_for_outbound_sockets=127.5.83.2
--webserver_interface=127.5.83.2
--webserver_port=0
--tserver_master_addrs=127.5.83.62:36133
--builtin_ntp_servers=127.5.83.20:35603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260808 04:04:27.484871 12158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:04:27.485128 12158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:04:27.485221 12158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:04:27.492214 12158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:04:27.492300 12158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:04:27.492434 12158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.2
I20260808 04:04:27.501037 12158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:35603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161826605655-5452-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161826605655-5452-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.83.2
--webserver_port=0
--tserver_master_addrs=127.5.83.62:36133
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.2
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161826605655-5452-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:04:27.502115 12158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:04:27.503695 12158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:04:27.515832 12165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:27.515878 12166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:27.515792 12169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:04:27.516917 12158 server_base.cc:1034] running on GCE node
I20260808 04:04:27.517469 12158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:04:27.518345 12158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:04:27.519562 12158 hybrid_clock.cc:648] HybridClock initialized: now 1786161867519511 us; error 34 us; skew 500 ppm
I20260808 04:04:27.519896 12158 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 04:04:27.521919 12158 webserver.cc:460] Webserver started at http://127.5.83.2:45343/ using document root <none> and password file <none>
I20260808 04:04:27.522536 12158 fs_manager.cc:359] Metadata directory not provided
I20260808 04:04:27.522639 12158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:04:27.522946 12158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:04:27.525657 12158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161826605655-5452-0/minicluster-data/ts-1/data/instance:
uuid: "20a86fe3c609400f98272153d3bf214b"
format_stamp: "Formatted at 2026-08-08 04:04:27 on dist-test-slave-1l3l"
server_key: "9f8145780c0a6825c23b04c74b9944c5"
server_key_iv: "72907fb8192869659697c44f036bdf0f"
server_key_version: "encryptionkey@0"
I20260808 04:04:27.526568 12158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161826605655-5452-0/minicluster-data/ts-1/wal/instance:
uuid: "20a86fe3c609400f98272153d3bf214b"
format_stamp: "Formatted at 2026-08-08 04:04:27 on dist-test-slave-1l3l"
server_key: "9f8145780c0a6825c23b04c74b9944c5"
server_key_iv: "72907fb8192869659697c44f036bdf0f"
server_key_version: "encryptionkey@0"
I20260808 04:04:27.531448 12158 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260808 04:04:27.534596 12176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:27.535924 12158 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:04:27.536072 12158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161826605655-5452-0/minicluster-data/ts-1/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161826605655-5452-0/minicluster-data/ts-1/wal
uuid: "20a86fe3c609400f98272153d3bf214b"
format_stamp: "Formatted at 2026-08-08 04:04:27 on dist-test-slave-1l3l"
server_key: "9f8145780c0a6825c23b04c74b9944c5"
server_key_iv: "72907fb8192869659697c44f036bdf0f"
server_key_version: "encryptionkey@0"
I20260808 04:04:27.536286 12158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161826605655-5452-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161826605655-5452-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161826605655-5452-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:04:27.541412 12158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:04:27.542115 12158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:04:27.542366 12158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:04:27.543192 12158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:04:27.544541 12158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:04:27.544641 12158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:27.544744 12158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:04:27.544812 12158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:27.573824 12158 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.2:43927
I20260808 04:04:27.573897 12329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.2:43927 every 8 connection(s)
I20260808 04:04:27.574998 12158 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
I20260808 04:04:27.576934 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 12158
I20260808 04:04:27.577090 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161826605655-5452-0/minicluster-data/ts-1/wal/instance
I20260808 04:04:27.577595 5452 external_mini_cluster.cc:1428] Setting key b5ab6f522620420fe8112eed61b36eef
I20260808 04:04:27.580639 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161826605655-5452-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161826605655-5452-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161826605655-5452-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.3:0
--local_ip_for_outbound_sockets=127.5.83.3
--webserver_interface=127.5.83.3
--webserver_port=0
--tserver_master_addrs=127.5.83.62:36133
--builtin_ntp_servers=127.5.83.20:35603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260808 04:04:27.583400 12154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:27.577472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.2:35885 (local address 127.5.83.62:36133)
0808 04:04:27.577682 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:27.577688 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:27.579092 (+ 1404us) server_negotiation.cc:407] Connection header received
0808 04:04:27.581261 (+ 2169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:27.581265 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:27.581334 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:27.581409 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:27.582772 (+ 1363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:27.582777 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:27.582780 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:04:27.582795 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:27.582829 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:04:27.582842 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:27.582927 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:27.583104 (+ 177us) server_negotiation.cc:299] Negotiation successful
0808 04:04:27.583149 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260808 04:04:27.584484 12333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:27.577984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:36133 (local address 127.5.83.2:35885)
0808 04:04:27.578953 (+ 969us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:27.578986 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:27.580979 (+ 1993us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:27.581665 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:27.581675 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:27.581786 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:04:27.582452 (+ 666us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:04:27.582455 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:04:27.582506 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:04:27.582607 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:04:27.582926 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:04:27.582935 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:04:27.582941 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 04:04:27.583189 (+ 248us) client_negotiation.cc:240] Negotiation successful
0808 04:04:27.583418 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":347,"threads_started":1}
I20260808 04:04:27.585667 12331 heartbeater.cc:347] Connected to a master server at 127.5.83.62:36133
I20260808 04:04:27.585951 12331 heartbeater.cc:464] Registering TS with master...
I20260808 04:04:27.586584 12331 heartbeater.cc:511] Master 127.5.83.62:36133 requested a full tablet report, sending...
I20260808 04:04:27.587947 11902 ts_manager.cc:194] Registered new tserver with Master: 20a86fe3c609400f98272153d3bf214b (127.5.83.2:43927)
W20260808 04:04:27.896142 12334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:04:27.896397 12334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:04:27.896533 12334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:04:27.903545 12334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:04:27.903630 12334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:04:27.903791 12334 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.3
I20260808 04:04:27.912621 12334 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:35603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161826605655-5452-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161826605655-5452-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.83.3
--webserver_port=0
--tserver_master_addrs=127.5.83.62:36133
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.3
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161826605655-5452-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:04:27.913769 12334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:04:27.915553 12334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:04:27.927801 12340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:27.927762 12343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:27.927824 12345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:04:27.928759 12334 server_base.cc:1034] running on GCE node
I20260808 04:04:27.929275 12334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:04:27.930124 12334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:04:27.931318 12334 hybrid_clock.cc:648] HybridClock initialized: now 1786161867931274 us; error 32 us; skew 500 ppm
I20260808 04:04:27.931624 12334 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 04:04:27.933687 12334 webserver.cc:460] Webserver started at http://127.5.83.3:42685/ using document root <none> and password file <none>
I20260808 04:04:27.934300 12334 fs_manager.cc:359] Metadata directory not provided
I20260808 04:04:27.934402 12334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:04:27.934710 12334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:04:27.937502 12334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161826605655-5452-0/minicluster-data/ts-2/data/instance:
uuid: "76aaec2bd9be449691fe8dd29f1b94f2"
format_stamp: "Formatted at 2026-08-08 04:04:27 on dist-test-slave-1l3l"
server_key: "1edf46cdf49e270ec1bc057a06d1d917"
server_key_iv: "307880fab79ce999ab120595dfc8b194"
server_key_version: "encryptionkey@0"
I20260808 04:04:27.938233 12334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161826605655-5452-0/minicluster-data/ts-2/wal/instance:
uuid: "76aaec2bd9be449691fe8dd29f1b94f2"
format_stamp: "Formatted at 2026-08-08 04:04:27 on dist-test-slave-1l3l"
server_key: "1edf46cdf49e270ec1bc057a06d1d917"
server_key_iv: "307880fab79ce999ab120595dfc8b194"
server_key_version: "encryptionkey@0"
I20260808 04:04:27.943074 12334 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260808 04:04:27.946303 12356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:27.947480 12334 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:04:27.947636 12334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161826605655-5452-0/minicluster-data/ts-2/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161826605655-5452-0/minicluster-data/ts-2/wal
uuid: "76aaec2bd9be449691fe8dd29f1b94f2"
format_stamp: "Formatted at 2026-08-08 04:04:27 on dist-test-slave-1l3l"
server_key: "1edf46cdf49e270ec1bc057a06d1d917"
server_key_iv: "307880fab79ce999ab120595dfc8b194"
server_key_version: "encryptionkey@0"
I20260808 04:04:27.947921 12334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161826605655-5452-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161826605655-5452-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161826605655-5452-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:04:27.953275 12334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:04:27.953956 12334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:04:27.954171 12334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:04:27.954963 12334 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:04:27.956341 12334 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:04:27.956437 12334 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:27.956532 12334 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:04:27.956604 12334 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:27.985636 12334 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.3:35759
I20260808 04:04:27.985682 12514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.3:35759 every 8 connection(s)
I20260808 04:04:27.986951 12334 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
I20260808 04:04:27.990841 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 12334
I20260808 04:04:27.991020 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161826605655-5452-0/minicluster-data/ts-2/wal/instance
I20260808 04:04:27.991470 5452 external_mini_cluster.cc:1428] Setting key 34f56ce7deb40d24eb962f502cfbf33d
I20260808 04:04:27.994805 12154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:27.989393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.3:48343 (local address 127.5.83.62:36133)
0808 04:04:27.989611 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:27.989617 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:27.991034 (+ 1417us) server_negotiation.cc:407] Connection header received
0808 04:04:27.992700 (+ 1666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:27.992705 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:27.992766 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:27.992850 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:27.994102 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:27.994108 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:27.994113 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:04:27.994128 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:27.994163 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:04:27.994175 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:27.994280 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:27.994467 (+ 187us) server_negotiation.cc:299] Negotiation successful
0808 04:04:27.994531 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260808 04:04:27.995647 12518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:27.989931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:36133 (local address 127.5.83.3:48343)
0808 04:04:27.990809 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:27.990849 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:27.992434 (+ 1585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:27.993087 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:27.993095 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:27.993179 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:04:27.993813 (+ 634us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:04:27.993815 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:04:27.993851 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:04:27.993934 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:04:27.994264 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:04:27.994274 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:04:27.994282 (+ 8us) client_negotiation.cc:769] Sending connection context
0808 04:04:27.994536 (+ 254us) client_negotiation.cc:240] Negotiation successful
0808 04:04:27.994766 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":270,"threads_started":1}
I20260808 04:04:27.996762 12515 heartbeater.cc:347] Connected to a master server at 127.5.83.62:36133
I20260808 04:04:27.997045 12515 heartbeater.cc:464] Registering TS with master...
I20260808 04:04:27.997602 12515 heartbeater.cc:511] Master 127.5.83.62:36133 requested a full tablet report, sending...
I20260808 04:04:27.998836 11902 ts_manager.cc:194] Registered new tserver with Master: 76aaec2bd9be449691fe8dd29f1b94f2 (127.5.83.3:35759)
I20260808 04:04:28.006014 5452 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:04:28.012694 12154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:28.010350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46792 (local address 127.5.83.62:36133)
0808 04:04:28.010558 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:28.010564 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:28.011036 (+ 472us) server_negotiation.cc:407] Connection header received
0808 04:04:28.011169 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:28.011173 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:28.011225 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:28.011293 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:28.012183 (+ 890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:28.012188 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:28.012191 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:04:28.012206 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:28.012234 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:04:28.012245 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:28.012340 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:28.012394 (+ 54us) server_negotiation.cc:299] Negotiation successful
0808 04:04:28.012439 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260808 04:04:28.016518 11902 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46792:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:04:28.022996 11902 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:04:28.041007 12543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:28.034547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.2:43927 (local address 127.0.0.1:50198)
0808 04:04:28.036486 (+ 1939us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:28.036529 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:28.036672 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:28.038697 (+ 2025us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:28.038704 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:28.038796 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:04:28.039527 (+ 731us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:04:28.039529 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:04:28.039551 (+ 22us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:04:28.039614 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:04:28.040459 (+ 845us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:04:28.040470 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:04:28.040478 (+ 8us) client_negotiation.cc:769] Sending connection context
0808 04:04:28.040651 (+ 173us) client_negotiation.cc:240] Negotiation successful
0808 04:04:28.040716 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1480,"thread_start_us":2636,"threads_started":1}
I20260808 04:04:28.041108 12540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:28.034821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50198 (local address 127.5.83.2:43927)
0808 04:04:28.037146 (+ 2325us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:28.037155 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:28.037185 (+ 30us) server_negotiation.cc:407] Connection header received
0808 04:04:28.037258 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:28.037266 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:28.037521 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:28.037901 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:28.040131 (+ 2230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:28.040141 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:28.040155 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:04:28.040181 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:28.040319 (+ 138us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:04:28.040345 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:28.040517 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:28.040662 (+ 145us) server_negotiation.cc:299] Negotiation successful
0808 04:04:28.040760 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2168,"thread_start_us":2052,"threads_started":1}
I20260808 04:04:28.044019 12543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:28.038179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:46841 (local address 127.0.0.1:50026)
0808 04:04:28.041194 (+ 3015us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:28.041209 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:28.041298 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:28.041836 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:28.041840 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:28.041879 (+ 39us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:04:28.042429 (+ 550us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:04:28.042433 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:04:28.042450 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:04:28.042493 (+ 43us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:04:28.043555 (+ 1062us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:04:28.043559 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:04:28.043564 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 04:04:28.043667 (+ 103us) client_negotiation.cc:240] Negotiation successful
0808 04:04:28.043735 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2937,"thread_start_us":1972,"threads_started":1}
I20260808 04:04:28.044185 12538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:28.035251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.3:35759 (local address 127.0.0.1:49640)
0808 04:04:28.037442 (+ 2191us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:28.037460 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:28.037549 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:28.041797 (+ 4248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:28.041802 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:28.041825 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:04:28.042606 (+ 781us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:04:28.042609 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:04:28.042626 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:04:28.042671 (+ 45us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:04:28.043460 (+ 789us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:04:28.043465 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:04:28.043471 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 04:04:28.043640 (+ 169us) client_negotiation.cc:240] Negotiation successful
0808 04:04:28.043691 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2080,"thread_start_us":1024,"threads_started":1}
I20260808 04:04:28.044096 12544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:28.035999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49640 (local address 127.5.83.3:35759)
0808 04:04:28.040704 (+ 4705us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:28.040712 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:28.040741 (+ 29us) server_negotiation.cc:407] Connection header received
0808 04:04:28.040815 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:28.040823 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:28.041016 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:28.042196 (+ 1180us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:28.043238 (+ 1042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:28.043245 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:28.043255 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:04:28.043271 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:28.043319 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:04:28.043338 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:28.043504 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:28.043611 (+ 107us) server_negotiation.cc:299] Negotiation successful
0808 04:04:28.043743 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4546,"thread_start_us":4381,"threads_started":1}
I20260808 04:04:28.044986 12546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:28.037831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50026 (local address 127.5.83.1:46841)
0808 04:04:28.039948 (+ 2117us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:28.039956 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:28.041388 (+ 1432us) server_negotiation.cc:407] Connection header received
0808 04:04:28.041477 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:28.041484 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:28.041674 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:28.042165 (+ 491us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:28.043235 (+ 1070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:28.043242 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:28.043254 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:04:28.043271 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:28.043319 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:04:28.043338 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:28.043482 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:28.044705 (+ 1223us) server_negotiation.cc:299] Negotiation successful
0808 04:04:28.044783 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1955,"thread_start_us":1448,"threads_started":1}
I20260808 04:04:28.045593 12241 tablet_service.cc:1467] Processing CreateTablet for tablet d14736e7016048ac9bf94022a36082db (DEFAULT_TABLE table=test-table [id=dcee53525ec4447ea3af72de7c76b952]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:04:28.047123 12241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d14736e7016048ac9bf94022a36082db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:04:28.047487 12422 tablet_service.cc:1467] Processing CreateTablet for tablet d14736e7016048ac9bf94022a36082db (DEFAULT_TABLE table=test-table [id=dcee53525ec4447ea3af72de7c76b952]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:04:28.048651 12422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d14736e7016048ac9bf94022a36082db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:04:28.048290 12064 tablet_service.cc:1467] Processing CreateTablet for tablet d14736e7016048ac9bf94022a36082db (DEFAULT_TABLE table=test-table [id=dcee53525ec4447ea3af72de7c76b952]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:04:28.049546 12064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d14736e7016048ac9bf94022a36082db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:04:28.059370 12550 tablet_bootstrap.cc:492] T d14736e7016048ac9bf94022a36082db P 76aaec2bd9be449691fe8dd29f1b94f2: Bootstrap starting.
I20260808 04:04:28.062454 12550 tablet_bootstrap.cc:654] T d14736e7016048ac9bf94022a36082db P 76aaec2bd9be449691fe8dd29f1b94f2: Neither blocks nor log segments found. Creating new log.
I20260808 04:04:28.063113 12551 tablet_bootstrap.cc:492] T d14736e7016048ac9bf94022a36082db P dd3c0452629247f8b6279bed425b57b1: Bootstrap starting.
I20260808 04:04:28.064134 12550 log.cc:824] T d14736e7016048ac9bf94022a36082db P 76aaec2bd9be449691fe8dd29f1b94f2: Log is configured to *not* fsync() on all Append() calls
I20260808 04:04:28.064457 12552 tablet_bootstrap.cc:492] T d14736e7016048ac9bf94022a36082db P 20a86fe3c609400f98272153d3bf214b: Bootstrap starting.
I20260808 04:04:28.066776 12551 tablet_bootstrap.cc:654] T d14736e7016048ac9bf94022a36082db P dd3c0452629247f8b6279bed425b57b1: Neither blocks nor log segments found. Creating new log.
I20260808 04:04:28.067482 12552 tablet_bootstrap.cc:654] T d14736e7016048ac9bf94022a36082db P 20a86fe3c609400f98272153d3bf214b: Neither blocks nor log segments found. Creating new log.
I20260808 04:04:28.067857 12551 log.cc:824] T d14736e7016048ac9bf94022a36082db P dd3c0452629247f8b6279bed425b57b1: Log is configured to *not* fsync() on all Append() calls
I20260808 04:04:28.068604 12552 log.cc:824] T d14736e7016048ac9bf94022a36082db P 20a86fe3c609400f98272153d3bf214b: Log is configured to *not* fsync() on all Append() calls
I20260808 04:04:28.069509 12550 tablet_bootstrap.cc:492] T d14736e7016048ac9bf94022a36082db P 76aaec2bd9be449691fe8dd29f1b94f2: No bootstrap required, opened a new log
I20260808 04:04:28.069790 12550 ts_tablet_manager.cc:1397] T d14736e7016048ac9bf94022a36082db P 76aaec2bd9be449691fe8dd29f1b94f2: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260808 04:04:28.070756 12551 tablet_bootstrap.cc:492] T d14736e7016048ac9bf94022a36082db P dd3c0452629247f8b6279bed425b57b1: No bootstrap required, opened a new log
I20260808 04:04:28.071002 12551 ts_tablet_manager.cc:1397] T d14736e7016048ac9bf94022a36082db P dd3c0452629247f8b6279bed425b57b1: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260808 04:04:28.073716 12552 tablet_bootstrap.cc:492] T d14736e7016048ac9bf94022a36082db P 20a86fe3c609400f98272153d3bf214b: No bootstrap required, opened a new log
I20260808 04:04:28.074015 12552 ts_tablet_manager.cc:1397] T d14736e7016048ac9bf94022a36082db P 20a86fe3c609400f98272153d3bf214b: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.005s
I20260808 04:04:28.079833 12550 raft_consensus.cc:348] T d14736e7016048ac9bf94022a36082db P 76aaec2bd9be449691fe8dd29f1b94f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a86fe3c609400f98272153d3bf214b" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 43927 } } peers { permanent_uuid: "dd3c0452629247f8b6279bed425b57b1" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 46841 } } peers { permanent_uuid: "76aaec2bd9be449691fe8dd29f1b94f2" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 35759 } }
I20260808 04:04:28.080258 12550 raft_consensus.cc:374] T d14736e7016048ac9bf94022a36082db P 76aaec2bd9be449691fe8dd29f1b94f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:04:28.080364 12550 raft_consensus.cc:729] T d14736e7016048ac9bf94022a36082db P 76aaec2bd9be449691fe8dd29f1b94f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76aaec2bd9be449691fe8dd29f1b94f2, State: Initialized, Role: FOLLOWER
I20260808 04:04:28.080790 12552 raft_consensus.cc:348] T d14736e7016048ac9bf94022a36082db P 20a86fe3c609400f98272153d3bf214b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a86fe3c609400f98272153d3bf214b" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 43927 } } peers { permanent_uuid: "dd3c0452629247f8b6279bed425b57b1" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 46841 } } peers { permanent_uuid: "76aaec2bd9be449691fe8dd29f1b94f2" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 35759 } }
I20260808 04:04:28.080808 12551 raft_consensus.cc:348] T d14736e7016048ac9bf94022a36082db P dd3c0452629247f8b6279bed425b57b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a86fe3c609400f98272153d3bf214b" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 43927 } } peers { permanent_uuid: "dd3c0452629247f8b6279bed425b57b1" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 46841 } } peers { permanent_uuid: "76aaec2bd9be449691fe8dd29f1b94f2" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 35759 } }
I20260808 04:04:28.081105 12552 raft_consensus.cc:374] T d14736e7016048ac9bf94022a36082db P 20a86fe3c609400f98272153d3bf214b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:04:28.080991 12550 consensus_queue.cc:260] T d14736e7016048ac9bf94022a36082db P 76aaec2bd9be449691fe8dd29f1b94f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a86fe3c609400f98272153d3bf214b" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 43927 } } peers { permanent_uuid: "dd3c0452629247f8b6279bed425b57b1" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 46841 } } peers { permanent_uuid: "76aaec2bd9be449691fe8dd29f1b94f2" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 35759 } }
I20260808 04:04:28.081244 12551 raft_consensus.cc:374] T d14736e7016048ac9bf94022a36082db P dd3c0452629247f8b6279bed425b57b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:04:28.081235 12552 raft_consensus.cc:729] T d14736e7016048ac9bf94022a36082db P 20a86fe3c609400f98272153d3bf214b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a86fe3c609400f98272153d3bf214b, State: Initialized, Role: FOLLOWER
I20260808 04:04:28.081377 12551 raft_consensus.cc:729] T d14736e7016048ac9bf94022a36082db P dd3c0452629247f8b6279bed425b57b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd3c0452629247f8b6279bed425b57b1, State: Initialized, Role: FOLLOWER
I20260808 04:04:28.081928 12552 consensus_queue.cc:260] T d14736e7016048ac9bf94022a36082db P 20a86fe3c609400f98272153d3bf214b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a86fe3c609400f98272153d3bf214b" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 43927 } } peers { permanent_uuid: "dd3c0452629247f8b6279bed425b57b1" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 46841 } } peers { permanent_uuid: "76aaec2bd9be449691fe8dd29f1b94f2" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 35759 } }
I20260808 04:04:28.082123 12515 heartbeater.cc:503] Master 127.5.83.62:36133 was elected leader, sending a full tablet report...
I20260808 04:04:28.082108 12551 consensus_queue.cc:260] T d14736e7016048ac9bf94022a36082db P dd3c0452629247f8b6279bed425b57b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a86fe3c609400f98272153d3bf214b" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 43927 } } peers { permanent_uuid: "dd3c0452629247f8b6279bed425b57b1" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 46841 } } peers { permanent_uuid: "76aaec2bd9be449691fe8dd29f1b94f2" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 35759 } }
I20260808 04:04:28.082813 12550 ts_tablet_manager.cc:1428] T d14736e7016048ac9bf94022a36082db P 76aaec2bd9be449691fe8dd29f1b94f2: Time spent starting tablet: real 0.013s user 0.009s sys 0.006s
I20260808 04:04:28.084169 12152 heartbeater.cc:503] Master 127.5.83.62:36133 was elected leader, sending a full tablet report...
I20260808 04:04:28.084424 12552 ts_tablet_manager.cc:1428] T d14736e7016048ac9bf94022a36082db P 20a86fe3c609400f98272153d3bf214b: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260808 04:04:28.084527 12331 heartbeater.cc:503] Master 127.5.83.62:36133 was elected leader, sending a full tablet report...
I20260808 04:04:28.085878 12551 ts_tablet_manager.cc:1428] T d14736e7016048ac9bf94022a36082db P dd3c0452629247f8b6279bed425b57b1: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260808 04:04:28.096206 12560 raft_consensus.cc:482] T d14736e7016048ac9bf94022a36082db P dd3c0452629247f8b6279bed425b57b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:04:28.096424 12560 raft_consensus.cc:504] T d14736e7016048ac9bf94022a36082db P dd3c0452629247f8b6279bed425b57b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a86fe3c609400f98272153d3bf214b" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 43927 } } peers { permanent_uuid: "dd3c0452629247f8b6279bed425b57b1" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 46841 } } peers { permanent_uuid: "76aaec2bd9be449691fe8dd29f1b94f2" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 35759 } }
I20260808 04:04:28.097868 12560 leader_election.cc:288] T d14736e7016048ac9bf94022a36082db P dd3c0452629247f8b6279bed425b57b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20a86fe3c609400f98272153d3bf214b (127.5.83.2:43927), 76aaec2bd9be449691fe8dd29f1b94f2 (127.5.83.3:35759)
I20260808 04:04:28.101294 12540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:28.098173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.1:51213 (local address 127.5.83.2:43927)
0808 04:04:28.098344 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:28.098350 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:28.099251 (+ 901us) server_negotiation.cc:407] Connection header received
0808 04:04:28.099444 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:28.099449 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:28.099528 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:28.099616 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:28.100723 (+ 1107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:28.100728 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:28.100732 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:04:28.100754 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:28.100789 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:04:28.100805 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:28.100922 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:28.100967 (+ 45us) server_negotiation.cc:299] Negotiation successful
0808 04:04:28.101038 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 04:04:28.101424 12544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:28.098226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.1:32885 (local address 127.5.83.3:35759)
0808 04:04:28.098384 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:28.098390 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:28.099271 (+ 881us) server_negotiation.cc:407] Connection header received
0808 04:04:28.099491 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:28.099497 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:28.099545 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:28.099628 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:28.100835 (+ 1207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:28.100839 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:28.100842 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:04:28.100856 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:28.100887 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:04:28.100897 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:28.101004 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:28.101092 (+ 88us) server_negotiation.cc:299] Negotiation successful
0808 04:04:28.101150 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 04:04:28.101536 12561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:28.098275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.2:43927 (local address 127.5.83.1:51213)
0808 04:04:28.099141 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:28.099163 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:28.099295 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:28.099655 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:28.099659 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:28.099681 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:04:28.100480 (+ 799us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:04:28.100484 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:04:28.100501 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:04:28.100546 (+ 45us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:04:28.100878 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:04:28.100883 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:04:28.100887 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 04:04:28.101020 (+ 133us) client_negotiation.cc:240] Negotiation successful
0808 04:04:28.101066 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":711,"thread_start_us":509,"threads_started":1}
I20260808 04:04:28.101665 12562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:28.098316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.3:35759 (local address 127.5.83.1:32885)
0808 04:04:28.099174 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:28.099192 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:28.099353 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:28.099667 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:28.099671 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:28.099692 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:04:28.100657 (+ 965us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:04:28.100660 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:04:28.100677 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:04:28.100717 (+ 40us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:04:28.100989 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:04:28.100998 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:04:28.101003 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 04:04:28.101139 (+ 136us) client_negotiation.cc:240] Negotiation successful
0808 04:04:28.101188 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":741,"thread_start_us":484,"threads_started":1}
I20260808 04:04:28.102437 12268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d14736e7016048ac9bf94022a36082db" candidate_uuid: "dd3c0452629247f8b6279bed425b57b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a86fe3c609400f98272153d3bf214b" is_pre_election: true
I20260808 04:04:28.102653 12455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d14736e7016048ac9bf94022a36082db" candidate_uuid: "dd3c0452629247f8b6279bed425b57b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76aaec2bd9be449691fe8dd29f1b94f2" is_pre_election: true
I20260808 04:04:28.102881 12268 raft_consensus.cc:2454] T d14736e7016048ac9bf94022a36082db P 20a86fe3c609400f98272153d3bf214b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd3c0452629247f8b6279bed425b57b1 in term 0.
I20260808 04:04:28.103163 12455 raft_consensus.cc:2454] T d14736e7016048ac9bf94022a36082db P 76aaec2bd9be449691fe8dd29f1b94f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd3c0452629247f8b6279bed425b57b1 in term 0.
I20260808 04:04:28.103447 12002 leader_election.cc:302] T d14736e7016048ac9bf94022a36082db P dd3c0452629247f8b6279bed425b57b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a86fe3c609400f98272153d3bf214b, dd3c0452629247f8b6279bed425b57b1; no voters:
I20260808 04:04:28.103821 12560 raft_consensus.cc:2781] T d14736e7016048ac9bf94022a36082db P dd3c0452629247f8b6279bed425b57b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:04:28.103976 12560 raft_consensus.cc:482] T d14736e7016048ac9bf94022a36082db P dd3c0452629247f8b6279bed425b57b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:04:28.104082 12560 raft_consensus.cc:3037] T d14736e7016048ac9bf94022a36082db P dd3c0452629247f8b6279bed425b57b1 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:04:28.106240 12560 raft_consensus.cc:504] T d14736e7016048ac9bf94022a36082db P dd3c0452629247f8b6279bed425b57b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a86fe3c609400f98272153d3bf214b" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 43927 } } peers { permanent_uuid: "dd3c0452629247f8b6279bed425b57b1" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 46841 } } peers { permanent_uuid: "76aaec2bd9be449691fe8dd29f1b94f2" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 35759 } }
I20260808 04:04:28.106767 12560 leader_election.cc:288] T d14736e7016048ac9bf94022a36082db P dd3c0452629247f8b6279bed425b57b1 [CANDIDATE]: Term 1 election: Requested vote from peers 20a86fe3c609400f98272153d3bf214b (127.5.83.2:43927), 76aaec2bd9be449691fe8dd29f1b94f2 (127.5.83.3:35759)
I20260808 04:04:28.107149 12268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d14736e7016048ac9bf94022a36082db" candidate_uuid: "dd3c0452629247f8b6279bed425b57b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a86fe3c609400f98272153d3bf214b"
I20260808 04:04:28.107213 12455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d14736e7016048ac9bf94022a36082db" candidate_uuid: "dd3c0452629247f8b6279bed425b57b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76aaec2bd9be449691fe8dd29f1b94f2"
I20260808 04:04:28.107369 12455 raft_consensus.cc:3037] T d14736e7016048ac9bf94022a36082db P 76aaec2bd9be449691fe8dd29f1b94f2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:04:28.107350 12268 raft_consensus.cc:3037] T d14736e7016048ac9bf94022a36082db P 20a86fe3c609400f98272153d3bf214b [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:04:28.110466 12455 raft_consensus.cc:2454] T d14736e7016048ac9bf94022a36082db P 76aaec2bd9be449691fe8dd29f1b94f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd3c0452629247f8b6279bed425b57b1 in term 1.
I20260808 04:04:28.110469 12268 raft_consensus.cc:2454] T d14736e7016048ac9bf94022a36082db P 20a86fe3c609400f98272153d3bf214b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd3c0452629247f8b6279bed425b57b1 in term 1.
I20260808 04:04:28.110832 12002 leader_election.cc:302] T d14736e7016048ac9bf94022a36082db P dd3c0452629247f8b6279bed425b57b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 20a86fe3c609400f98272153d3bf214b, 76aaec2bd9be449691fe8dd29f1b94f2, dd3c0452629247f8b6279bed425b57b1; no voters:
I20260808 04:04:28.111135 12560 raft_consensus.cc:2781] T d14736e7016048ac9bf94022a36082db P dd3c0452629247f8b6279bed425b57b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:04:28.111569 12560 raft_consensus.cc:686] T d14736e7016048ac9bf94022a36082db P dd3c0452629247f8b6279bed425b57b1 [term 1 LEADER]: Becoming Leader. State: Replica: dd3c0452629247f8b6279bed425b57b1, State: Running, Role: LEADER
I20260808 04:04:28.112066 12560 consensus_queue.cc:237] T d14736e7016048ac9bf94022a36082db P dd3c0452629247f8b6279bed425b57b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a86fe3c609400f98272153d3bf214b" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 43927 } } peers { permanent_uuid: "dd3c0452629247f8b6279bed425b57b1" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 46841 } } peers { permanent_uuid: "76aaec2bd9be449691fe8dd29f1b94f2" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 35759 } }
I20260808 04:04:28.116154 11899 catalog_manager.cc:5495] T d14736e7016048ac9bf94022a36082db P dd3c0452629247f8b6279bed425b57b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to dd3c0452629247f8b6279bed425b57b1 (127.5.83.1). New cstate: current_term: 1 leader_uuid: "dd3c0452629247f8b6279bed425b57b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a86fe3c609400f98272153d3bf214b" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 43927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd3c0452629247f8b6279bed425b57b1" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 46841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76aaec2bd9be449691fe8dd29f1b94f2" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 35759 } health_report { overall_health: UNKNOWN } } }
I20260808 04:04:28.144110 12546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:28.142129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50040 (local address 127.5.83.1:46841)
0808 04:04:28.142283 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:28.142288 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:28.142426 (+ 138us) server_negotiation.cc:407] Connection header received
0808 04:04:28.142623 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:28.142626 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:28.142705 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:28.142779 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:28.143581 (+ 802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:28.143585 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:28.143588 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:04:28.143600 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:28.143627 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:04:28.143637 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:28.143749 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:28.143843 (+ 94us) server_negotiation.cc:299] Negotiation successful
0808 04:04:28.143898 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 04:04:28.151806 12268 raft_consensus.cc:1261] T d14736e7016048ac9bf94022a36082db P 20a86fe3c609400f98272153d3bf214b [term 1 FOLLOWER]: Refusing update from remote peer dd3c0452629247f8b6279bed425b57b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:04:28.151928 12455 raft_consensus.cc:1261] T d14736e7016048ac9bf94022a36082db P 76aaec2bd9be449691fe8dd29f1b94f2 [term 1 FOLLOWER]: Refusing update from remote peer dd3c0452629247f8b6279bed425b57b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:04:28.152830 12564 consensus_queue.cc:1035] T d14736e7016048ac9bf94022a36082db P dd3c0452629247f8b6279bed425b57b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76aaec2bd9be449691fe8dd29f1b94f2" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 35759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:04:28.153193 12560 consensus_queue.cc:1035] T d14736e7016048ac9bf94022a36082db P dd3c0452629247f8b6279bed425b57b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a86fe3c609400f98272153d3bf214b" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 43927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 04:04:28.154912 12153 tablet.cc:2369] T d14736e7016048ac9bf94022a36082db P dd3c0452629247f8b6279bed425b57b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:04:28.165084 12567 mvcc.cc:205] Tried to move back new op lower bound from 7316119011940446208 to 7316119011790311424. Current Snapshot: MvccSnapshot[applied={T|T < 7316119011940446208}]
I20260808 04:04:28.177673 12570 mvcc.cc:205] Tried to move back new op lower bound from 7316119011940446208 to 7316119011790311424. Current Snapshot: MvccSnapshot[applied={T|T < 7316119011940446208}]
I20260808 04:04:28.183081 11899 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46792:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:04:28.183311 11899 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46792:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:04:28.186646 11899 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7da96244aa394e2b942371d4dce3a145: Sending DeleteTablet for 3 replicas of tablet d14736e7016048ac9bf94022a36082db
I20260808 04:04:28.187731 12241 tablet_service.cc:1514] Processing DeleteTablet for tablet d14736e7016048ac9bf94022a36082db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:04:28 UTC) from {username='slave'} at 127.0.0.1:50198
I20260808 04:04:28.187788 12422 tablet_service.cc:1514] Processing DeleteTablet for tablet d14736e7016048ac9bf94022a36082db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:04:28 UTC) from {username='slave'} at 127.0.0.1:49640
I20260808 04:04:28.188685 12064 tablet_service.cc:1514] Processing DeleteTablet for tablet d14736e7016048ac9bf94022a36082db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:04:28 UTC) from {username='slave'} at 127.0.0.1:50026
I20260808 04:04:28.188943 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 11982
I20260808 04:04:28.196736 12580 tablet_replica.cc:330] T d14736e7016048ac9bf94022a36082db P 76aaec2bd9be449691fe8dd29f1b94f2: stopping tablet replica
I20260808 04:04:28.196884 12581 tablet_replica.cc:330] T d14736e7016048ac9bf94022a36082db P dd3c0452629247f8b6279bed425b57b1: stopping tablet replica
I20260808 04:04:28.197029 12579 tablet_replica.cc:330] T d14736e7016048ac9bf94022a36082db P 20a86fe3c609400f98272153d3bf214b: stopping tablet replica
I20260808 04:04:28.197170 12580 raft_consensus.cc:2229] T d14736e7016048ac9bf94022a36082db P 76aaec2bd9be449691fe8dd29f1b94f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:04:28.197271 12581 raft_consensus.cc:2229] T d14736e7016048ac9bf94022a36082db P dd3c0452629247f8b6279bed425b57b1 [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:04:28.197456 12579 raft_consensus.cc:2229] T d14736e7016048ac9bf94022a36082db P 20a86fe3c609400f98272153d3bf214b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:04:28.197662 12580 raft_consensus.cc:2258] T d14736e7016048ac9bf94022a36082db P 76aaec2bd9be449691fe8dd29f1b94f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:04:28.197979 12579 raft_consensus.cc:2258] T d14736e7016048ac9bf94022a36082db P 20a86fe3c609400f98272153d3bf214b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:04:28.197945 12581 raft_consensus.cc:2258] T d14736e7016048ac9bf94022a36082db P dd3c0452629247f8b6279bed425b57b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:04:28.200132 12579 ts_tablet_manager.cc:1905] T d14736e7016048ac9bf94022a36082db P 20a86fe3c609400f98272153d3bf214b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:04:28.200205 12581 ts_tablet_manager.cc:1905] T d14736e7016048ac9bf94022a36082db P dd3c0452629247f8b6279bed425b57b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:04:28.200716 12546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:28.190324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50054 (local address 127.5.83.1:46841)
0808 04:04:28.190502 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:28.190509 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:28.196242 (+ 5733us) server_negotiation.cc:407] Connection header received
0808 04:04:28.197122 (+ 880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:28.197127 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:28.197188 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:28.197265 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:28.199467 (+ 2202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:28.199473 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:28.199477 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:04:28.199496 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:28.199531 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:04:28.199546 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:28.199682 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:28.200346 (+ 664us) server_negotiation.cc:299] Negotiation successful
0808 04:04:28.200411 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 04:04:28.201830 12146 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:04:28.202298 12580 ts_tablet_manager.cc:1905] T d14736e7016048ac9bf94022a36082db P 76aaec2bd9be449691fe8dd29f1b94f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:04:28.206929 12579 ts_tablet_manager.cc:1918] T d14736e7016048ac9bf94022a36082db P 20a86fe3c609400f98272153d3bf214b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:04:28.207082 12579 log.cc:1196] T d14736e7016048ac9bf94022a36082db P 20a86fe3c609400f98272153d3bf214b: Deleting WAL directory at /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161826605655-5452-0/minicluster-data/ts-1/wal/wals/d14736e7016048ac9bf94022a36082db
I20260808 04:04:28.207597 12579 ts_tablet_manager.cc:1939] T d14736e7016048ac9bf94022a36082db P 20a86fe3c609400f98272153d3bf214b: Deleting consensus metadata
I20260808 04:04:28.208986 11876 catalog_manager.cc:4841] TS 20a86fe3c609400f98272153d3bf214b (127.5.83.2:43927): tablet d14736e7016048ac9bf94022a36082db (table test-table [id=dcee53525ec4447ea3af72de7c76b952]) successfully deleted
I20260808 04:04:28.209343 12580 ts_tablet_manager.cc:1918] T d14736e7016048ac9bf94022a36082db P 76aaec2bd9be449691fe8dd29f1b94f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:04:28.209568 12580 log.cc:1196] T d14736e7016048ac9bf94022a36082db P 76aaec2bd9be449691fe8dd29f1b94f2: Deleting WAL directory at /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161826605655-5452-0/minicluster-data/ts-2/wal/wals/d14736e7016048ac9bf94022a36082db
I20260808 04:04:28.210124 12580 ts_tablet_manager.cc:1939] T d14736e7016048ac9bf94022a36082db P 76aaec2bd9be449691fe8dd29f1b94f2: Deleting consensus metadata
I20260808 04:04:28.211778 11880 catalog_manager.cc:4841] TS 76aaec2bd9be449691fe8dd29f1b94f2 (127.5.83.3:35759): tablet d14736e7016048ac9bf94022a36082db (table test-table [id=dcee53525ec4447ea3af72de7c76b952]) successfully deleted
I20260808 04:04:28.296792 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 11982
I20260808 04:04:28.317246 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 12158
W20260808 04:04:28.317080 11876 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.83.1:46841 (error 108)
I20260808 04:04:28.319092 12538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:28.318546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:46841 (local address 127.0.0.1:60454)
0808 04:04:28.318733 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:28.318955 (+ 222us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.1:46841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260808 04:04:28.319341 11876 catalog_manager.cc:4584] TS dd3c0452629247f8b6279bed425b57b1 (127.5.83.1:46841): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d14736e7016048ac9bf94022a36082db: Network error: Client connection negotiation failed: client connection to 127.5.83.1:46841: connect: Connection refused (error 111)
I20260808 04:04:28.320542 12540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:28.318058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60230 (local address 127.5.83.2:43927)
0808 04:04:28.318240 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:28.318246 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:28.318345 (+ 99us) server_negotiation.cc:407] Connection header received
0808 04:04:28.318538 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:28.318542 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:28.318647 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:28.318747 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:28.319955 (+ 1208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:28.319960 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:28.319963 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:04:28.319978 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:28.320009 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:04:28.320021 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:28.320134 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:28.320225 (+ 91us) server_negotiation.cc:299] Negotiation successful
0808 04:04:28.320285 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 04:04:28.321126 12324 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:04:28.372797 12538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:28.372336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:46841 (local address 127.0.0.1:60460)
0808 04:04:28.372532 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:28.372632 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.1:46841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260808 04:04:28.373766 12538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:28.373464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:46841 (local address 127.0.0.1:60468)
0808 04:04:28.373580 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:28.373651 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.1:46841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260808 04:04:28.408185 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 12158
I20260808 04:04:28.428467 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 12334
I20260808 04:04:28.431072 12544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:28.429282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34454 (local address 127.5.83.3:35759)
0808 04:04:28.429491 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:28.429495 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:28.429514 (+ 19us) server_negotiation.cc:407] Connection header received
0808 04:04:28.429573 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:28.429576 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:28.429667 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:28.429751 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:28.430555 (+ 804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:28.430559 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:28.430563 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:04:28.430578 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:28.430611 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:04:28.430624 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:28.430723 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:28.430777 (+ 54us) server_negotiation.cc:299] Negotiation successful
0808 04:04:28.430838 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260808 04:04:28.431699 12510 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:04:28.434585 12538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:28.434155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:46841 (local address 127.0.0.1:60476)
0808 04:04:28.434337 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:28.434451 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.1:46841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260808 04:04:28.435926 12538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:28.435503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:46841 (local address 127.0.0.1:60480)
0808 04:04:28.435651 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:28.435782 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.1:46841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260808 04:04:28.516743 12538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:28.516246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:46841 (local address 127.0.0.1:60490)
0808 04:04:28.516435 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:28.516542 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.1:46841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260808 04:04:28.517925 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 12334
I20260808 04:04:28.518306 12538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:28.517830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:46841 (local address 127.0.0.1:60492)
0808 04:04:28.518072 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:28.518160 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.1:46841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260808 04:04:28.537822 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 11851
I20260808 04:04:28.538997 11938 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:04:28.644524 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 11851
2026-08-08T04:04:28Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2498 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260808 04:04:28.688347 5452 mini_oidc.cc:152] Starting JWKS server
I20260808 04:04:28.691164 5452 webserver.cc:460] Webserver started at https://127.0.0.1:42159/ using document root <none> and password file <none>
I20260808 04:04:28.691439 5452 mini_oidc.cc:189] Starting OIDC Discovery server
I20260808 04:04:28.692047 5452 webserver.cc:460] Webserver started at http://127.0.0.1:44901/ using document root <none> and password file <none>
2026-08-08T04:04:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:04:28Z Disabled control of system clock
I20260808 04:04:28.720423 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786161826605655-5452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786161826605655-5452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786161826605655-5452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:36399
--webserver_interface=127.5.83.62
--webserver_port=0
--builtin_ntp_servers=127.5.83.20:45759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.62:36399
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42159/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786161826605655-5452-0/testchainca.cert with env {}
W20260808 04:04:29.031828 12598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:04:29.032097 12598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:04:29.032188 12598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:04:29.039283 12598 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:04:29.039378 12598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:04:29.039420 12598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:04:29.039474 12598 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:04:29.039517 12598 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260808 04:04:29.039575 12598 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260808 04:04:29.039613 12598 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42159/jwks.json
I20260808 04:04:29.049378 12598 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:45759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786161826605655-5452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786161826605655-5452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.62:36399
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:36399
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42159/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786161826605655-5452-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786161826605655-5452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:04:29.050546 12598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:04:29.051767 12598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:04:29.062567 12604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:29.062595 12606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:29.062762 12608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:04:29.063478 12598 server_base.cc:1034] running on GCE node
I20260808 04:04:29.064185 12598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:04:29.065414 12598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:04:29.066605 12598 hybrid_clock.cc:648] HybridClock initialized: now 1786161869066556 us; error 43 us; skew 500 ppm
I20260808 04:04:29.066969 12598 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 04:04:29.069161 12598 webserver.cc:460] Webserver started at http://127.5.83.62:41155/ using document root <none> and password file <none>
I20260808 04:04:29.069794 12598 fs_manager.cc:359] Metadata directory not provided
I20260808 04:04:29.069895 12598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:04:29.070184 12598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:04:29.073077 12598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786161826605655-5452-0/minicluster-data/master-0/data/instance:
uuid: "a626223456aa4e15a5c1a3395ff6cae0"
format_stamp: "Formatted at 2026-08-08 04:04:29 on dist-test-slave-1l3l"
server_key: "bc8c6a9793565a19a86fe3314f009637"
server_key_iv: "15366be7b55b84198e2f8aebe7454022"
server_key_version: "encryptionkey@0"
I20260808 04:04:29.073869 12598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786161826605655-5452-0/minicluster-data/master-0/wal/instance:
uuid: "a626223456aa4e15a5c1a3395ff6cae0"
format_stamp: "Formatted at 2026-08-08 04:04:29 on dist-test-slave-1l3l"
server_key: "bc8c6a9793565a19a86fe3314f009637"
server_key_iv: "15366be7b55b84198e2f8aebe7454022"
server_key_version: "encryptionkey@0"
I20260808 04:04:29.078853 12598 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260808 04:04:29.082154 12616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:29.083401 12598 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 04:04:29.083566 12598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786161826605655-5452-0/minicluster-data/master-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786161826605655-5452-0/minicluster-data/master-0/wal
uuid: "a626223456aa4e15a5c1a3395ff6cae0"
format_stamp: "Formatted at 2026-08-08 04:04:29 on dist-test-slave-1l3l"
server_key: "bc8c6a9793565a19a86fe3314f009637"
server_key_iv: "15366be7b55b84198e2f8aebe7454022"
server_key_version: "encryptionkey@0"
I20260808 04:04:29.083804 12598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786161826605655-5452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786161826605655-5452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786161826605655-5452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:04:29.100594 12598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:04:29.101332 12598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:04:29.101588 12598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:04:29.118455 12598 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.62:36399
I20260808 04:04:29.118496 12686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.62:36399 every 8 connection(s)
I20260808 04:04:29.119686 12598 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
I20260808 04:04:29.123188 12689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:04:29.127287 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 12598
I20260808 04:04:29.127526 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786161826605655-5452-0/minicluster-data/master-0/wal/instance
I20260808 04:04:29.128367 5452 external_mini_cluster.cc:1428] Setting key 96a640bdb97c70338245c91b652abc1d
I20260808 04:04:29.134306 12689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a626223456aa4e15a5c1a3395ff6cae0: Bootstrap starting.
I20260808 04:04:29.137745 12689 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a626223456aa4e15a5c1a3395ff6cae0: Neither blocks nor log segments found. Creating new log.
I20260808 04:04:29.138960 12689 log.cc:824] T 00000000000000000000000000000000 P a626223456aa4e15a5c1a3395ff6cae0: Log is configured to *not* fsync() on all Append() calls
I20260808 04:04:29.141752 12692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:29.130188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33862 (local address 127.5.83.62:36399)
0808 04:04:29.131077 (+ 889us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:29.131088 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:29.131130 (+ 42us) server_negotiation.cc:407] Connection header received
0808 04:04:29.132909 (+ 1779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:29.132931 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:29.133337 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:29.133637 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:29.134266 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:29.135567 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:29.137417 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:29.138226 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:29.138354 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:29.139578 (+ 1224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:29.139585 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:29.139598 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:04:29.140155 (+ 557us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:29.140228 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:04:29.140248 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:29.140352 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:29.140559 (+ 207us) server_negotiation.cc:299] Negotiation successful
0808 04:04:29.140830 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":317,"threads_started":1}
I20260808 04:04:29.141773 12689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a626223456aa4e15a5c1a3395ff6cae0: No bootstrap required, opened a new log
I20260808 04:04:29.145164 5452 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:04:29.149317 12689 raft_consensus.cc:348] T 00000000000000000000000000000000 P a626223456aa4e15a5c1a3395ff6cae0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a626223456aa4e15a5c1a3395ff6cae0" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 36399 } }
I20260808 04:04:29.149641 12689 raft_consensus.cc:374] T 00000000000000000000000000000000 P a626223456aa4e15a5c1a3395ff6cae0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:04:29.149729 12689 raft_consensus.cc:729] T 00000000000000000000000000000000 P a626223456aa4e15a5c1a3395ff6cae0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a626223456aa4e15a5c1a3395ff6cae0, State: Initialized, Role: FOLLOWER
I20260808 04:04:29.150249 12689 consensus_queue.cc:260] T 00000000000000000000000000000000 P a626223456aa4e15a5c1a3395ff6cae0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a626223456aa4e15a5c1a3395ff6cae0" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 36399 } }
I20260808 04:04:29.150461 12689 raft_consensus.cc:388] T 00000000000000000000000000000000 P a626223456aa4e15a5c1a3395ff6cae0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:04:29.150585 12689 raft_consensus.cc:482] T 00000000000000000000000000000000 P a626223456aa4e15a5c1a3395ff6cae0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:04:29.150736 12689 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a626223456aa4e15a5c1a3395ff6cae0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:04:29.153004 12689 raft_consensus.cc:504] T 00000000000000000000000000000000 P a626223456aa4e15a5c1a3395ff6cae0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a626223456aa4e15a5c1a3395ff6cae0" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 36399 } }
I20260808 04:04:29.153470 12689 leader_election.cc:302] T 00000000000000000000000000000000 P a626223456aa4e15a5c1a3395ff6cae0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a626223456aa4e15a5c1a3395ff6cae0; no voters:
I20260808 04:04:29.154163 12689 leader_election.cc:288] T 00000000000000000000000000000000 P a626223456aa4e15a5c1a3395ff6cae0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:04:29.154282 12695 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a626223456aa4e15a5c1a3395ff6cae0 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:04:29.154994 12695 raft_consensus.cc:686] T 00000000000000000000000000000000 P a626223456aa4e15a5c1a3395ff6cae0 [term 1 LEADER]: Becoming Leader. State: Replica: a626223456aa4e15a5c1a3395ff6cae0, State: Running, Role: LEADER
I20260808 04:04:29.155573 12695 consensus_queue.cc:237] T 00000000000000000000000000000000 P a626223456aa4e15a5c1a3395ff6cae0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a626223456aa4e15a5c1a3395ff6cae0" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 36399 } }
I20260808 04:04:29.156379 12689 sys_catalog.cc:564] T 00000000000000000000000000000000 P a626223456aa4e15a5c1a3395ff6cae0 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:04:29.158237 12699 sys_catalog.cc:455] T 00000000000000000000000000000000 P a626223456aa4e15a5c1a3395ff6cae0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a626223456aa4e15a5c1a3395ff6cae0. Latest consensus state: current_term: 1 leader_uuid: "a626223456aa4e15a5c1a3395ff6cae0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a626223456aa4e15a5c1a3395ff6cae0" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 36399 } } }
I20260808 04:04:29.158653 12699 sys_catalog.cc:458] T 00000000000000000000000000000000 P a626223456aa4e15a5c1a3395ff6cae0 [sys.catalog]: This master's current role is: LEADER
I20260808 04:04:29.159502 12696 sys_catalog.cc:455] T 00000000000000000000000000000000 P a626223456aa4e15a5c1a3395ff6cae0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a626223456aa4e15a5c1a3395ff6cae0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a626223456aa4e15a5c1a3395ff6cae0" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 36399 } } }
I20260808 04:04:29.159919 12696 sys_catalog.cc:458] T 00000000000000000000000000000000 P a626223456aa4e15a5c1a3395ff6cae0 [sys.catalog]: This master's current role is: LEADER
I20260808 04:04:29.160924 12711 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:04:29.165392 12711 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:04:29.172154 12724 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260808 04:04:29.173467 12711 catalog_manager.cc:1347] Generated new cluster ID: a78c59a7a32f4fc7a75ba111bfce1740
I20260808 04:04:29.173568 12711 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:04:29.191314 12711 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:04:29.193068 12711 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:04:29.204840 12711 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a626223456aa4e15a5c1a3395ff6cae0: Generated new TSK 0
I20260808 04:04:29.205818 12711 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260808 04:04:29.253383 12739 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260808 04:04:29.253573 12739 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260808 04:04:29.254442 12692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:29.226680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33866 (local address 127.5.83.62:36399)
0808 04:04:29.226879 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:29.226886 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:29.227233 (+ 347us) server_negotiation.cc:407] Connection header received
0808 04:04:29.227554 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:29.227558 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:29.227635 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:29.227836 (+ 201us) server_negotiation.cc:226] Negotiated authn=JWT
0808 04:04:29.228390 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:29.229137 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:29.230598 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:29.231061 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:29.231173 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:29.231739 (+ 566us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0808 04:04:29.253772 (+ 22033us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42159/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0808 04:04:29.254138 (+ 366us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33866: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42159/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":46}
W20260808 04:04:29.254578 12692 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33866: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42159/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260808 04:04:29.256263 12737 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:04:29.226515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:36399 (local address 127.0.0.1:33866)
0808 04:04:29.227125 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:29.227145 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:29.227332 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:29.227844 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:29.227849 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:29.227861 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0808 04:04:29.228199 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:29.228210 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:29.229291 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:29.229297 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:29.230452 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:29.230464 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:29.231187 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:29.231254 (+ 67us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:29.231394 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:29.231448 (+ 54us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0808 04:04:29.254041 (+ 22593us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42159/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0808 04:04:29.254375 (+ 334us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.5.83.62:36399: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42159/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":387,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.5.83.62:36399: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42159/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260808 04:04:29.257766 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 12598
I20260808 04:04:29.258922 12682 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:04:29.350991 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 12598
2026-08-08T04:04:29Z chronyd exiting
I20260808 04:04:29.404912 5452 test_util.cc:175] -----------------------------------------------
I20260808 04:04:29.405045 5452 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786161826605655-5452-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (721 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260808 04:04:29.408860 5452 mini_oidc.cc:152] Starting JWKS server
I20260808 04:04:29.410677 5452 webserver.cc:460] Webserver started at https://127.0.0.1:42217/ using document root <none> and password file <none>
I20260808 04:04:29.410899 5452 mini_oidc.cc:189] Starting OIDC Discovery server
I20260808 04:04:29.411525 5452 webserver.cc:460] Webserver started at http://127.0.0.1:38187/ using document root <none> and password file <none>
2026-08-08T04:04:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:04:29Z Disabled control of system clock
I20260808 04:04:29.437973 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786161826605655-5452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786161826605655-5452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786161826605655-5452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:34497
--webserver_interface=127.5.83.62
--webserver_port=0
--builtin_ntp_servers=127.5.83.20:39245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.62:34497
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42217/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786161826605655-5452-0/testchainca.cert with env {}
W20260808 04:04:29.744855 12748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:04:29.745105 12748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:04:29.745229 12748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:04:29.752028 12748 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:04:29.752115 12748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:04:29.752174 12748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:04:29.752245 12748 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:04:29.752303 12748 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260808 04:04:29.752393 12748 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260808 04:04:29.752444 12748 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42217/jwks.json
I20260808 04:04:29.761668 12748 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:39245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786161826605655-5452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786161826605655-5452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.62:34497
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:34497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42217/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786161826605655-5452-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786161826605655-5452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:04:29.762764 12748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:04:29.763979 12748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:04:29.774706 12759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:29.774781 12757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:29.774750 12761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:04:29.775758 12748 server_base.cc:1034] running on GCE node
I20260808 04:04:29.776458 12748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:04:29.777750 12748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:04:29.778957 12748 hybrid_clock.cc:648] HybridClock initialized: now 1786161869778906 us; error 44 us; skew 500 ppm
I20260808 04:04:29.779273 12748 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 04:04:29.781370 12748 webserver.cc:460] Webserver started at http://127.5.83.62:35875/ using document root <none> and password file <none>
I20260808 04:04:29.782020 12748 fs_manager.cc:359] Metadata directory not provided
I20260808 04:04:29.782128 12748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:04:29.782529 12748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:04:29.785507 12748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786161826605655-5452-0/minicluster-data/master-0/data/instance:
uuid: "ffc277ac954547bb8e8fb1c98e305b59"
format_stamp: "Formatted at 2026-08-08 04:04:29 on dist-test-slave-1l3l"
server_key: "d6ca2c6d2b473a6c56d7d9eadaa7b522"
server_key_iv: "faeb4b09e854d603243b10686e55338f"
server_key_version: "encryptionkey@0"
I20260808 04:04:29.786270 12748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786161826605655-5452-0/minicluster-data/master-0/wal/instance:
uuid: "ffc277ac954547bb8e8fb1c98e305b59"
format_stamp: "Formatted at 2026-08-08 04:04:29 on dist-test-slave-1l3l"
server_key: "d6ca2c6d2b473a6c56d7d9eadaa7b522"
server_key_iv: "faeb4b09e854d603243b10686e55338f"
server_key_version: "encryptionkey@0"
I20260808 04:04:29.790989 12748 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260808 04:04:29.794196 12770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:29.795439 12748 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:04:29.795598 12748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786161826605655-5452-0/minicluster-data/master-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786161826605655-5452-0/minicluster-data/master-0/wal
uuid: "ffc277ac954547bb8e8fb1c98e305b59"
format_stamp: "Formatted at 2026-08-08 04:04:29 on dist-test-slave-1l3l"
server_key: "d6ca2c6d2b473a6c56d7d9eadaa7b522"
server_key_iv: "faeb4b09e854d603243b10686e55338f"
server_key_version: "encryptionkey@0"
I20260808 04:04:29.795830 12748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786161826605655-5452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786161826605655-5452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786161826605655-5452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:04:29.811012 12748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:04:29.811866 12748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:04:29.812137 12748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:04:29.828295 12748 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.62:34497
I20260808 04:04:29.828303 12840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.62:34497 every 8 connection(s)
I20260808 04:04:29.829584 12748 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
I20260808 04:04:29.833120 12841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:04:29.833503 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 12748
I20260808 04:04:29.833675 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786161826605655-5452-0/minicluster-data/master-0/wal/instance
I20260808 04:04:29.834299 5452 external_mini_cluster.cc:1428] Setting key fce00647016d10467cfdf3c0f08d9f08
I20260808 04:04:29.846980 12841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffc277ac954547bb8e8fb1c98e305b59: Bootstrap starting.
I20260808 04:04:29.848959 12845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:29.836125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53788 (local address 127.5.83.62:34497)
0808 04:04:29.837178 (+ 1053us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:29.837193 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:29.837247 (+ 54us) server_negotiation.cc:407] Connection header received
0808 04:04:29.839401 (+ 2154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:29.839427 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:29.840016 (+ 589us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:29.840445 (+ 429us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:29.840976 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:29.842787 (+ 1811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:29.844094 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:29.845121 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:29.845294 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:29.846319 (+ 1025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:29.846329 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:29.846345 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:04:29.847033 (+ 688us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:29.847125 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:04:29.847150 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:29.847258 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:29.847456 (+ 198us) server_negotiation.cc:299] Negotiation successful
0808 04:04:29.847770 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":335,"threads_started":1}
I20260808 04:04:29.850559 12841 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ffc277ac954547bb8e8fb1c98e305b59: Neither blocks nor log segments found. Creating new log.
I20260808 04:04:29.851725 12841 log.cc:824] T 00000000000000000000000000000000 P ffc277ac954547bb8e8fb1c98e305b59: Log is configured to *not* fsync() on all Append() calls
I20260808 04:04:29.852690 5452 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:04:29.854322 12841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffc277ac954547bb8e8fb1c98e305b59: No bootstrap required, opened a new log
I20260808 04:04:29.861302 12841 raft_consensus.cc:348] T 00000000000000000000000000000000 P ffc277ac954547bb8e8fb1c98e305b59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffc277ac954547bb8e8fb1c98e305b59" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 34497 } }
I20260808 04:04:29.861603 12841 raft_consensus.cc:374] T 00000000000000000000000000000000 P ffc277ac954547bb8e8fb1c98e305b59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:04:29.861690 12841 raft_consensus.cc:729] T 00000000000000000000000000000000 P ffc277ac954547bb8e8fb1c98e305b59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffc277ac954547bb8e8fb1c98e305b59, State: Initialized, Role: FOLLOWER
I20260808 04:04:29.862200 12841 consensus_queue.cc:260] T 00000000000000000000000000000000 P ffc277ac954547bb8e8fb1c98e305b59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffc277ac954547bb8e8fb1c98e305b59" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 34497 } }
I20260808 04:04:29.862408 12841 raft_consensus.cc:388] T 00000000000000000000000000000000 P ffc277ac954547bb8e8fb1c98e305b59 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:04:29.862529 12841 raft_consensus.cc:482] T 00000000000000000000000000000000 P ffc277ac954547bb8e8fb1c98e305b59 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:04:29.862669 12841 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ffc277ac954547bb8e8fb1c98e305b59 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:04:29.864676 12841 raft_consensus.cc:504] T 00000000000000000000000000000000 P ffc277ac954547bb8e8fb1c98e305b59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffc277ac954547bb8e8fb1c98e305b59" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 34497 } }
I20260808 04:04:29.865113 12841 leader_election.cc:302] T 00000000000000000000000000000000 P ffc277ac954547bb8e8fb1c98e305b59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ffc277ac954547bb8e8fb1c98e305b59; no voters:
I20260808 04:04:29.865625 12841 leader_election.cc:288] T 00000000000000000000000000000000 P ffc277ac954547bb8e8fb1c98e305b59 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:04:29.865737 12848 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ffc277ac954547bb8e8fb1c98e305b59 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:04:29.866580 12848 raft_consensus.cc:686] T 00000000000000000000000000000000 P ffc277ac954547bb8e8fb1c98e305b59 [term 1 LEADER]: Becoming Leader. State: Replica: ffc277ac954547bb8e8fb1c98e305b59, State: Running, Role: LEADER
I20260808 04:04:29.867146 12848 consensus_queue.cc:237] T 00000000000000000000000000000000 P ffc277ac954547bb8e8fb1c98e305b59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffc277ac954547bb8e8fb1c98e305b59" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 34497 } }
I20260808 04:04:29.867319 12841 sys_catalog.cc:564] T 00000000000000000000000000000000 P ffc277ac954547bb8e8fb1c98e305b59 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:04:29.869884 12851 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffc277ac954547bb8e8fb1c98e305b59 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ffc277ac954547bb8e8fb1c98e305b59. Latest consensus state: current_term: 1 leader_uuid: "ffc277ac954547bb8e8fb1c98e305b59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffc277ac954547bb8e8fb1c98e305b59" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 34497 } } }
I20260808 04:04:29.869957 12850 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffc277ac954547bb8e8fb1c98e305b59 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ffc277ac954547bb8e8fb1c98e305b59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffc277ac954547bb8e8fb1c98e305b59" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 34497 } } }
I20260808 04:04:29.870317 12850 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffc277ac954547bb8e8fb1c98e305b59 [sys.catalog]: This master's current role is: LEADER
I20260808 04:04:29.870316 12851 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffc277ac954547bb8e8fb1c98e305b59 [sys.catalog]: This master's current role is: LEADER
I20260808 04:04:29.871142 12857 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:04:29.876781 12857 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:04:29.884280 12857 catalog_manager.cc:1347] Generated new cluster ID: c1202ef5ad3a4b92857c0e09eef92c4e
I20260808 04:04:29.884367 12857 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:04:29.893174 12857 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:04:29.894289 12857 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:04:29.909339 12857 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ffc277ac954547bb8e8fb1c98e305b59: Generated new TSK 0
I20260808 04:04:29.910145 12857 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260808 04:04:29.946836 12879 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260808 04:04:29.946957 12879 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260808 04:04:29.947921 12845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:29.929113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53804 (local address 127.5.83.62:34497)
0808 04:04:29.929291 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:29.929297 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:29.929690 (+ 393us) server_negotiation.cc:407] Connection header received
0808 04:04:29.929887 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:29.929890 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:29.929960 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:29.930072 (+ 112us) server_negotiation.cc:226] Negotiated authn=JWT
0808 04:04:29.930578 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:29.931272 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:29.933280 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:29.933718 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:29.933805 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:29.934318 (+ 513us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0808 04:04:29.947371 (+ 13053us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42217/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0808 04:04:29.947661 (+ 290us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53804: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42217/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":61}
W20260808 04:04:29.948040 12845 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53804: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42217/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260808 04:04:29.948228 12878 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:04:29.928943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:34497 (local address 127.0.0.1:53804)
0808 04:04:29.929587 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:29.929607 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:29.929750 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:29.930110 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:29.930115 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:29.930129 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0808 04:04:29.930440 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:29.930452 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:29.931413 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:29.931418 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:29.933147 (+ 1729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:29.933162 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:29.933824 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:29.933870 (+ 46us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:29.934066 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:29.934109 (+ 43us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0808 04:04:29.947556 (+ 13447us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42217/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0808 04:04:29.947848 (+ 292us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.5.83.62:34497: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42217/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":321,"threads_started":1}
I20260808 04:04:29.949820 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 12748
I20260808 04:04:29.950778 12834 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:04:30.040442 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 12748
2026-08-08T04:04:30Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (687 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260808 04:04:30.095988 5452 mini_oidc.cc:152] Starting JWKS server
I20260808 04:04:30.097793 5452 webserver.cc:460] Webserver started at https://127.0.0.1:38893/ using document root <none> and password file <none>
I20260808 04:04:30.098022 5452 mini_oidc.cc:189] Starting OIDC Discovery server
I20260808 04:04:30.098639 5452 webserver.cc:460] Webserver started at http://127.0.0.1:38269/ using document root <none> and password file <none>
2026-08-08T04:04:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:04:30Z Disabled control of system clock
I20260808 04:04:30.127200 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786161826605655-5452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786161826605655-5452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786161826605655-5452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:37395
--webserver_interface=127.5.83.62
--webserver_port=0
--builtin_ntp_servers=127.5.83.20:45921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.62:37395
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38893/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260808 04:04:30.437147 12890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:04:30.437407 12890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:04:30.437520 12890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:04:30.444590 12890 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:04:30.444689 12890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:04:30.444728 12890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:04:30.444804 12890 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:04:30.444851 12890 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260808 04:04:30.444942 12890 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260808 04:04:30.444993 12890 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38893/jwks.json
I20260808 04:04:30.453850 12890 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:45921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786161826605655-5452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786161826605655-5452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.62:37395
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:37395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38893/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786161826605655-5452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:04:30.455039 12890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:04:30.456794 12890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:04:30.467545 12905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:30.467497 12899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:30.467576 12900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:04:30.468389 12890 server_base.cc:1034] running on GCE node
I20260808 04:04:30.469077 12890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:04:30.470283 12890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:04:30.471487 12890 hybrid_clock.cc:648] HybridClock initialized: now 1786161870471441 us; error 41 us; skew 500 ppm
I20260808 04:04:30.471822 12890 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 04:04:30.477109 12890 webserver.cc:460] Webserver started at http://127.5.83.62:41893/ using document root <none> and password file <none>
I20260808 04:04:30.477725 12890 fs_manager.cc:359] Metadata directory not provided
I20260808 04:04:30.477823 12890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:04:30.478277 12890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:04:30.481164 12890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786161826605655-5452-0/minicluster-data/master-0/data/instance:
uuid: "f9ec61b4e2974b02b1e2d7fa4a39f9f6"
format_stamp: "Formatted at 2026-08-08 04:04:30 on dist-test-slave-1l3l"
server_key: "9547ceb71a0fc8158e6816b067519a14"
server_key_iv: "ec1c0ab5093d7c61c99bc29db27d510e"
server_key_version: "encryptionkey@0"
I20260808 04:04:30.481940 12890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786161826605655-5452-0/minicluster-data/master-0/wal/instance:
uuid: "f9ec61b4e2974b02b1e2d7fa4a39f9f6"
format_stamp: "Formatted at 2026-08-08 04:04:30 on dist-test-slave-1l3l"
server_key: "9547ceb71a0fc8158e6816b067519a14"
server_key_iv: "ec1c0ab5093d7c61c99bc29db27d510e"
server_key_version: "encryptionkey@0"
I20260808 04:04:30.486711 12890 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 04:04:30.489812 12914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:30.490998 12890 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260808 04:04:30.491149 12890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786161826605655-5452-0/minicluster-data/master-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786161826605655-5452-0/minicluster-data/master-0/wal
uuid: "f9ec61b4e2974b02b1e2d7fa4a39f9f6"
format_stamp: "Formatted at 2026-08-08 04:04:30 on dist-test-slave-1l3l"
server_key: "9547ceb71a0fc8158e6816b067519a14"
server_key_iv: "ec1c0ab5093d7c61c99bc29db27d510e"
server_key_version: "encryptionkey@0"
I20260808 04:04:30.491331 12890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786161826605655-5452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786161826605655-5452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786161826605655-5452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:04:30.501892 12890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:04:30.502568 12890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:04:30.502810 12890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:04:30.518893 12890 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.62:37395
I20260808 04:04:30.518923 12989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.62:37395 every 8 connection(s)
I20260808 04:04:30.520095 12890 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
I20260808 04:04:30.523069 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 12890
I20260808 04:04:30.523281 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786161826605655-5452-0/minicluster-data/master-0/wal/instance
I20260808 04:04:30.524009 12990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:04:30.524142 5452 external_mini_cluster.cc:1428] Setting key bf6de49d3025e23fa4423c9a4d7bb03e
I20260808 04:04:30.535633 12990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9ec61b4e2974b02b1e2d7fa4a39f9f6: Bootstrap starting.
I20260808 04:04:30.536345 12994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:30.525942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45274 (local address 127.5.83.62:37395)
0808 04:04:30.526878 (+ 936us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:30.526898 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:30.526937 (+ 39us) server_negotiation.cc:407] Connection header received
0808 04:04:30.528551 (+ 1614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:30.528572 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:30.528993 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:30.529325 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:30.529855 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:30.531096 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:30.532373 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:30.533040 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:30.533209 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:30.534265 (+ 1056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:30.534272 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:30.534283 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:04:30.534759 (+ 476us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:30.534827 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:04:30.534847 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:30.534948 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:30.535152 (+ 204us) server_negotiation.cc:299] Negotiation successful
0808 04:04:30.535435 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":345,"threads_started":1}
I20260808 04:04:30.538931 12990 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f9ec61b4e2974b02b1e2d7fa4a39f9f6: Neither blocks nor log segments found. Creating new log.
I20260808 04:04:30.539917 5452 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:04:30.540038 12990 log.cc:824] T 00000000000000000000000000000000 P f9ec61b4e2974b02b1e2d7fa4a39f9f6: Log is configured to *not* fsync() on all Append() calls
I20260808 04:04:30.542789 12990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9ec61b4e2974b02b1e2d7fa4a39f9f6: No bootstrap required, opened a new log
I20260808 04:04:30.549701 12990 raft_consensus.cc:348] T 00000000000000000000000000000000 P f9ec61b4e2974b02b1e2d7fa4a39f9f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9ec61b4e2974b02b1e2d7fa4a39f9f6" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 37395 } }
I20260808 04:04:30.550010 12990 raft_consensus.cc:374] T 00000000000000000000000000000000 P f9ec61b4e2974b02b1e2d7fa4a39f9f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:04:30.550102 12990 raft_consensus.cc:729] T 00000000000000000000000000000000 P f9ec61b4e2974b02b1e2d7fa4a39f9f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9ec61b4e2974b02b1e2d7fa4a39f9f6, State: Initialized, Role: FOLLOWER
I20260808 04:04:30.550562 12990 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9ec61b4e2974b02b1e2d7fa4a39f9f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9ec61b4e2974b02b1e2d7fa4a39f9f6" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 37395 } }
I20260808 04:04:30.550755 12990 raft_consensus.cc:388] T 00000000000000000000000000000000 P f9ec61b4e2974b02b1e2d7fa4a39f9f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:04:30.550848 12990 raft_consensus.cc:482] T 00000000000000000000000000000000 P f9ec61b4e2974b02b1e2d7fa4a39f9f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:04:30.550964 12990 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f9ec61b4e2974b02b1e2d7fa4a39f9f6 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:04:30.553194 12990 raft_consensus.cc:504] T 00000000000000000000000000000000 P f9ec61b4e2974b02b1e2d7fa4a39f9f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9ec61b4e2974b02b1e2d7fa4a39f9f6" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 37395 } }
I20260808 04:04:30.553617 12990 leader_election.cc:302] T 00000000000000000000000000000000 P f9ec61b4e2974b02b1e2d7fa4a39f9f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f9ec61b4e2974b02b1e2d7fa4a39f9f6; no voters:
I20260808 04:04:30.554114 12990 leader_election.cc:288] T 00000000000000000000000000000000 P f9ec61b4e2974b02b1e2d7fa4a39f9f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:04:30.554234 12999 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f9ec61b4e2974b02b1e2d7fa4a39f9f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:04:30.554831 12999 raft_consensus.cc:686] T 00000000000000000000000000000000 P f9ec61b4e2974b02b1e2d7fa4a39f9f6 [term 1 LEADER]: Becoming Leader. State: Replica: f9ec61b4e2974b02b1e2d7fa4a39f9f6, State: Running, Role: LEADER
I20260808 04:04:30.555330 12999 consensus_queue.cc:237] T 00000000000000000000000000000000 P f9ec61b4e2974b02b1e2d7fa4a39f9f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9ec61b4e2974b02b1e2d7fa4a39f9f6" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 37395 } }
I20260808 04:04:30.556102 12990 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9ec61b4e2974b02b1e2d7fa4a39f9f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:04:30.557818 13000 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9ec61b4e2974b02b1e2d7fa4a39f9f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f9ec61b4e2974b02b1e2d7fa4a39f9f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9ec61b4e2974b02b1e2d7fa4a39f9f6" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 37395 } } }
I20260808 04:04:30.558225 13000 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9ec61b4e2974b02b1e2d7fa4a39f9f6 [sys.catalog]: This master's current role is: LEADER
I20260808 04:04:30.558954 13001 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9ec61b4e2974b02b1e2d7fa4a39f9f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f9ec61b4e2974b02b1e2d7fa4a39f9f6. Latest consensus state: current_term: 1 leader_uuid: "f9ec61b4e2974b02b1e2d7fa4a39f9f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9ec61b4e2974b02b1e2d7fa4a39f9f6" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 37395 } } }
I20260808 04:04:30.559330 13001 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9ec61b4e2974b02b1e2d7fa4a39f9f6 [sys.catalog]: This master's current role is: LEADER
I20260808 04:04:30.560472 13008 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:04:30.565152 13008 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:04:30.574139 13008 catalog_manager.cc:1347] Generated new cluster ID: fd71ca82437a43b0a48f8e0f81acfc27
I20260808 04:04:30.574239 13008 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:04:30.585934 13008 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:04:30.587101 13008 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:04:30.594029 13008 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f9ec61b4e2974b02b1e2d7fa4a39f9f6: Generated new TSK 0
I20260808 04:04:30.594983 13008 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:04:30.621636 12994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:30.613456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45290 (local address 127.5.83.62:37395)
0808 04:04:30.613631 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:30.613637 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:30.614131 (+ 494us) server_negotiation.cc:407] Connection header received
0808 04:04:30.614312 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:30.614317 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:30.614423 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:30.614578 (+ 155us) server_negotiation.cc:226] Negotiated authn=JWT
0808 04:04:30.614956 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:30.616089 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:30.617423 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:30.618126 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:30.618269 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:30.618716 (+ 447us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0808 04:04:30.621002 (+ 2286us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38893/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0808 04:04:30.621322 (+ 320us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45290: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38893/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":53}
W20260808 04:04:30.621766 12994 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45290: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38893/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260808 04:04:30.623238 13035 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:04:30.613424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:37395 (local address 127.0.0.1:45290)
0808 04:04:30.614033 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:30.614053 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:30.614167 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:30.614580 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:30.614583 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:30.614592 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0808 04:04:30.614828 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:30.614835 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:30.616209 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:30.616213 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:30.617283 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:30.617293 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:30.618251 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:30.618314 (+ 63us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:30.618442 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:30.618474 (+ 32us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0808 04:04:30.621208 (+ 2734us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38893/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0808 04:04:30.622813 (+ 1605us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.5.83.62:37395: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38893/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":486,"mutex_wait_us":1329,"thread_start_us":317,"threads_started":1}
I20260808 04:04:30.624284 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 12890
I20260808 04:04:30.625367 12985 generic_service.cc:196] Checking for leaks (request via RPC)
E20260808 04:04:30.632848 13036 webserver.cc:533] Webserver: sslize failed:
E20260808 04:04:30.632936 13036 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260808 04:04:30.713660 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 12890
2026-08-08T04:04:30Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (675 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-08T04:04:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:04:30Z Disabled control of system clock
I20260808 04:04:30.802120 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786161826605655-5452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786161826605655-5452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786161826605655-5452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:38165
--webserver_interface=127.5.83.62
--webserver_port=0
--builtin_ntp_servers=127.5.83.20:36129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.62:38165
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786161826605655-5452-0/testchainca.cert with env {}
W20260808 04:04:31.117022 13044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:04:31.117303 13044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:04:31.117406 13044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:04:31.124322 13044 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:04:31.124426 13044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:04:31.124464 13044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:04:31.124518 13044 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:04:31.124562 13044 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260808 04:04:31.124621 13044 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260808 04:04:31.124660 13044 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260808 04:04:31.134569 13044 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:36129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786161826605655-5452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786161826605655-5452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.62:38165
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:38165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786161826605655-5452-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786161826605655-5452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:04:31.135936 13044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:04:31.137347 13044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:04:31.148653 13053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:31.148788 13052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:31.148788 13055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:04:31.149628 13044 server_base.cc:1034] running on GCE node
I20260808 04:04:31.150319 13044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:04:31.151793 13044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:04:31.153052 13044 hybrid_clock.cc:648] HybridClock initialized: now 1786161871153004 us; error 37 us; skew 500 ppm
I20260808 04:04:31.153383 13044 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 04:04:31.155589 13044 webserver.cc:460] Webserver started at http://127.5.83.62:34919/ using document root <none> and password file <none>
I20260808 04:04:31.156283 13044 fs_manager.cc:359] Metadata directory not provided
I20260808 04:04:31.156391 13044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:04:31.156689 13044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:04:31.159420 13044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786161826605655-5452-0/minicluster-data/master-0/data/instance:
uuid: "1242eb12bdea478ea20b3a3f7c1033b8"
format_stamp: "Formatted at 2026-08-08 04:04:31 on dist-test-slave-1l3l"
server_key: "41c78772e85fc9d8324ad1c0a53d8e3b"
server_key_iv: "8d74530d6d234a80f1cd2ca5d73f3be3"
server_key_version: "encryptionkey@0"
I20260808 04:04:31.160228 13044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786161826605655-5452-0/minicluster-data/master-0/wal/instance:
uuid: "1242eb12bdea478ea20b3a3f7c1033b8"
format_stamp: "Formatted at 2026-08-08 04:04:31 on dist-test-slave-1l3l"
server_key: "41c78772e85fc9d8324ad1c0a53d8e3b"
server_key_iv: "8d74530d6d234a80f1cd2ca5d73f3be3"
server_key_version: "encryptionkey@0"
I20260808 04:04:31.165020 13044 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 04:04:31.168280 13066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:31.169562 13044 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:04:31.169723 13044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786161826605655-5452-0/minicluster-data/master-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786161826605655-5452-0/minicluster-data/master-0/wal
uuid: "1242eb12bdea478ea20b3a3f7c1033b8"
format_stamp: "Formatted at 2026-08-08 04:04:31 on dist-test-slave-1l3l"
server_key: "41c78772e85fc9d8324ad1c0a53d8e3b"
server_key_iv: "8d74530d6d234a80f1cd2ca5d73f3be3"
server_key_version: "encryptionkey@0"
I20260808 04:04:31.169919 13044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786161826605655-5452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786161826605655-5452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786161826605655-5452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:04:31.187480 13044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:04:31.188257 13044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:04:31.188516 13044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:04:31.205143 13044 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.62:38165
I20260808 04:04:31.205161 13137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.62:38165 every 8 connection(s)
I20260808 04:04:31.206384 13044 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
I20260808 04:04:31.209930 13138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:04:31.210973 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 13044
I20260808 04:04:31.211176 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786161826605655-5452-0/minicluster-data/master-0/wal/instance
I20260808 04:04:31.212086 5452 external_mini_cluster.cc:1428] Setting key 6bedad58c275e3f21860fbea8f17a411
I20260808 04:04:31.223963 13138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1242eb12bdea478ea20b3a3f7c1033b8: Bootstrap starting.
I20260808 04:04:31.225610 13141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:31.213963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58792 (local address 127.5.83.62:38165)
0808 04:04:31.214924 (+ 961us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:31.214934 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:31.214981 (+ 47us) server_negotiation.cc:407] Connection header received
0808 04:04:31.216702 (+ 1721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:31.216723 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:31.217170 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:31.217540 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:31.218134 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:31.219376 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:31.221248 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:31.221970 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:31.222107 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:31.223392 (+ 1285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:31.223400 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:31.223413 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:04:31.223978 (+ 565us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:31.224056 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:04:31.224076 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:31.224180 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:31.224378 (+ 198us) server_negotiation.cc:299] Negotiation successful
0808 04:04:31.224675 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":355,"threads_started":1}
I20260808 04:04:31.227399 13138 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1242eb12bdea478ea20b3a3f7c1033b8: Neither blocks nor log segments found. Creating new log.
I20260808 04:04:31.228554 13138 log.cc:824] T 00000000000000000000000000000000 P 1242eb12bdea478ea20b3a3f7c1033b8: Log is configured to *not* fsync() on all Append() calls
I20260808 04:04:31.229233 5452 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:04:31.231132 13138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1242eb12bdea478ea20b3a3f7c1033b8: No bootstrap required, opened a new log
I20260808 04:04:31.237933 13138 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1242eb12bdea478ea20b3a3f7c1033b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1242eb12bdea478ea20b3a3f7c1033b8" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 38165 } }
I20260808 04:04:31.238251 13138 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1242eb12bdea478ea20b3a3f7c1033b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:04:31.238340 13138 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1242eb12bdea478ea20b3a3f7c1033b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1242eb12bdea478ea20b3a3f7c1033b8, State: Initialized, Role: FOLLOWER
I20260808 04:04:31.238823 13138 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1242eb12bdea478ea20b3a3f7c1033b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1242eb12bdea478ea20b3a3f7c1033b8" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 38165 } }
I20260808 04:04:31.239037 13138 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1242eb12bdea478ea20b3a3f7c1033b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:04:31.239156 13138 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1242eb12bdea478ea20b3a3f7c1033b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:04:31.239301 13138 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1242eb12bdea478ea20b3a3f7c1033b8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:04:31.241422 13138 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1242eb12bdea478ea20b3a3f7c1033b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1242eb12bdea478ea20b3a3f7c1033b8" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 38165 } }
I20260808 04:04:31.241859 13138 leader_election.cc:302] T 00000000000000000000000000000000 P 1242eb12bdea478ea20b3a3f7c1033b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1242eb12bdea478ea20b3a3f7c1033b8; no voters:
I20260808 04:04:31.242460 13138 leader_election.cc:288] T 00000000000000000000000000000000 P 1242eb12bdea478ea20b3a3f7c1033b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:04:31.242573 13143 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1242eb12bdea478ea20b3a3f7c1033b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:04:31.243229 13143 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1242eb12bdea478ea20b3a3f7c1033b8 [term 1 LEADER]: Becoming Leader. State: Replica: 1242eb12bdea478ea20b3a3f7c1033b8, State: Running, Role: LEADER
I20260808 04:04:31.243781 13143 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1242eb12bdea478ea20b3a3f7c1033b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1242eb12bdea478ea20b3a3f7c1033b8" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 38165 } }
I20260808 04:04:31.244376 13138 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1242eb12bdea478ea20b3a3f7c1033b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:04:31.246294 13146 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1242eb12bdea478ea20b3a3f7c1033b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1242eb12bdea478ea20b3a3f7c1033b8. Latest consensus state: current_term: 1 leader_uuid: "1242eb12bdea478ea20b3a3f7c1033b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1242eb12bdea478ea20b3a3f7c1033b8" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 38165 } } }
I20260808 04:04:31.246498 13144 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1242eb12bdea478ea20b3a3f7c1033b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1242eb12bdea478ea20b3a3f7c1033b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1242eb12bdea478ea20b3a3f7c1033b8" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 38165 } } }
I20260808 04:04:31.246778 13146 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1242eb12bdea478ea20b3a3f7c1033b8 [sys.catalog]: This master's current role is: LEADER
I20260808 04:04:31.246930 13144 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1242eb12bdea478ea20b3a3f7c1033b8 [sys.catalog]: This master's current role is: LEADER
I20260808 04:04:31.248498 13149 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:04:31.252671 13149 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:04:31.261046 13149 catalog_manager.cc:1347] Generated new cluster ID: a4d3381bcf3548ea8c66f880ae07961b
I20260808 04:04:31.261140 13149 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:04:31.281220 13149 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:04:31.282941 13149 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:04:31.292469 13149 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1242eb12bdea478ea20b3a3f7c1033b8: Generated new TSK 0
I20260808 04:04:31.293444 13149 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 04:04:31.326136 13176 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:04:31.309742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:38165 (local address 127.0.0.1:58804)
0808 04:04:31.310381 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:31.310401 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:31.310554 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:31.310950 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:31.310955 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:31.310968 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0808 04:04:31.311310 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:31.311323 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:31.312346 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:31.312352 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:31.314041 (+ 1689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:31.314076 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:31.314736 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:31.314790 (+ 54us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:31.314979 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:31.315023 (+ 44us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0808 04:04:31.325395 (+ 10372us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0808 04:04:31.325691 (+ 296us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.5.83.62:38165: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":351,"threads_started":1}
I20260808 04:04:31.326700 13141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:31.309867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58804 (local address 127.5.83.62:38165)
0808 04:04:31.310088 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:31.310098 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:31.310488 (+ 390us) server_negotiation.cc:407] Connection header received
0808 04:04:31.310714 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:31.310718 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:31.310781 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:31.310914 (+ 133us) server_negotiation.cc:226] Negotiated authn=JWT
0808 04:04:31.311452 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:31.312208 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:31.314193 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:31.314629 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:31.314714 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:31.315243 (+ 529us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0808 04:04:31.325137 (+ 9894us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0808 04:04:31.326394 (+ 1257us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58804: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":78}
W20260808 04:04:31.326921 13141 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58804: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
I20260808 04:04:31.327147 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 13044
I20260808 04:04:31.328311 13131 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:04:31.419063 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 13044
2026-08-08T04:04:31Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (684 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (394 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (393 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:04:32 dist-test-slave-1l3l krb5kdc[13198](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:04:32 dist-test-slave-1l3l krb5kdc[13198](info): set up 2 sockets
Aug 08 04:04:32 dist-test-slave-1l3l krb5kdc[13198](info): commencing operation
krb5kdc: starting...
W20260808 04:04:33.374346 5452 mini_kdc.cc:121] Time spent starting KDC: real 1.130s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/krb5kdc/test-user.keytab.
Aug 08 04:04:33 dist-test-slave-1l3l krb5kdc[13198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:04:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:04:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.83.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab.
Entry for principal kudu/127.5.83.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab.
I20260808 04:04:33.566865 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:37771
--webserver_interface=127.5.83.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab
--principal=kudu/127.5.83.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:43547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.62:37771
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:04:33.874601 13216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:04:33.874878 13216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:04:33.875018 13216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:04:33.882023 13216 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:04:33.882130 13216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:04:33.882215 13216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:04:33.882292 13216 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:04:33.882360 13216 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:04:33.891747 13216 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:43547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.62:37771
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab
--principal=kudu/127.5.83.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:37771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:04:33.892939 13216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:04:33.894225 13216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:04:33.905265 13229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:33.905296 13228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:33.905615 13235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:04:33.905994 13216 server_base.cc:1034] running on GCE node
I20260808 04:04:33.906687 13216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:04:33.907960 13216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:04:33.909169 13216 hybrid_clock.cc:648] HybridClock initialized: now 1786161873909116 us; error 37 us; skew 500 ppm
I20260808 04:04:33.909499 13216 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:04:33 dist-test-slave-1l3l krb5kdc[13198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161873, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:04:33.912456 13216 init.cc:377] Logged in from keytab as kudu/127.5.83.62@KRBTEST.COM (short username kudu)
I20260808 04:04:33.913728 13216 webserver.cc:460] Webserver started at http://127.5.83.62:34145/ using document root <none> and password file <none>
I20260808 04:04:33.914322 13216 fs_manager.cc:359] Metadata directory not provided
I20260808 04:04:33.914420 13216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:04:33.914677 13216 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:04:33.917477 13216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/minicluster-data/master-0/data/instance:
uuid: "4f7e4478de0f4abeae2f26fa2648b328"
format_stamp: "Formatted at 2026-08-08 04:04:33 on dist-test-slave-1l3l"
server_key: "9b048a012d3703d28c1c3626193c7672"
server_key_iv: "82f10d9d20da6d45ea4c3cf3238ddd9d"
server_key_version: "encryptionkey@0"
I20260808 04:04:33.918195 13216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/minicluster-data/master-0/wal/instance:
uuid: "4f7e4478de0f4abeae2f26fa2648b328"
format_stamp: "Formatted at 2026-08-08 04:04:33 on dist-test-slave-1l3l"
server_key: "9b048a012d3703d28c1c3626193c7672"
server_key_iv: "82f10d9d20da6d45ea4c3cf3238ddd9d"
server_key_version: "encryptionkey@0"
I20260808 04:04:33.923204 13216 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260808 04:04:33.926443 13248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:33.927631 13216 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:04:33.927798 13216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/minicluster-data/master-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/minicluster-data/master-0/wal
uuid: "4f7e4478de0f4abeae2f26fa2648b328"
format_stamp: "Formatted at 2026-08-08 04:04:33 on dist-test-slave-1l3l"
server_key: "9b048a012d3703d28c1c3626193c7672"
server_key_iv: "82f10d9d20da6d45ea4c3cf3238ddd9d"
server_key_version: "encryptionkey@0"
I20260808 04:04:33.927975 13216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:04:33.945672 13216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:04:33.949472 13216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:04:33.949688 13216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:04:33.965859 13216 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.62:37771
I20260808 04:04:33.965902 13315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.62:37771 every 8 connection(s)
I20260808 04:04:33.967061 13216 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
I20260808 04:04:33.970633 13316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:04:33.973366 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 13216
I20260808 04:04:33.973537 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/minicluster-data/master-0/wal/instance
I20260808 04:04:33.974156 5452 external_mini_cluster.cc:1428] Setting key b12ea02b071d29f8a6361c0c33165c58
I20260808 04:04:33.982916 13316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f7e4478de0f4abeae2f26fa2648b328: Bootstrap starting.
I20260808 04:04:33.986280 13316 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f7e4478de0f4abeae2f26fa2648b328: Neither blocks nor log segments found. Creating new log.
I20260808 04:04:33.987213 13316 log.cc:824] T 00000000000000000000000000000000 P 4f7e4478de0f4abeae2f26fa2648b328: Log is configured to *not* fsync() on all Append() calls
Aug 08 04:04:33 dist-test-slave-1l3l krb5kdc[13198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:04:33.989948 13316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f7e4478de0f4abeae2f26fa2648b328: No bootstrap required, opened a new log
I20260808 04:04:33.996127 13321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:33.975983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48894 (local address 127.5.83.62:37771)
0808 04:04:33.976779 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:33.976793 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:33.976852 (+ 59us) server_negotiation.cc:407] Connection header received
0808 04:04:33.978862 (+ 2010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:33.978888 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:33.979364 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:33.979791 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:33.981017 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:33.982917 (+ 1900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:33.984149 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:33.985160 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:33.985348 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:33.989280 (+ 3932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:33.989287 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:33.989300 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:33.989333 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:33.992262 (+ 2929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:33.993040 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:33.993049 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:33.993055 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:33.993206 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:33.993590 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:33.993594 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:33.993595 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:33.994082 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:33.994306 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:33.994728 (+ 422us) server_negotiation.cc:299] Negotiation successful
0808 04:04:33.995028 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":351,"threads_started":1}
I20260808 04:04:33.998087 13316 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f7e4478de0f4abeae2f26fa2648b328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f7e4478de0f4abeae2f26fa2648b328" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 37771 } }
I20260808 04:04:33.998413 13316 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4f7e4478de0f4abeae2f26fa2648b328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:04:33.998513 13316 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f7e4478de0f4abeae2f26fa2648b328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f7e4478de0f4abeae2f26fa2648b328, State: Initialized, Role: FOLLOWER
I20260808 04:04:33.999063 13316 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f7e4478de0f4abeae2f26fa2648b328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f7e4478de0f4abeae2f26fa2648b328" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 37771 } }
I20260808 04:04:33.999270 13316 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4f7e4478de0f4abeae2f26fa2648b328 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:04:33.999396 13316 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4f7e4478de0f4abeae2f26fa2648b328 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:04:33.999545 13316 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f7e4478de0f4abeae2f26fa2648b328 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:04:34.001549 13316 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4f7e4478de0f4abeae2f26fa2648b328 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f7e4478de0f4abeae2f26fa2648b328" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 37771 } }
I20260808 04:04:34.001983 13316 leader_election.cc:302] T 00000000000000000000000000000000 P 4f7e4478de0f4abeae2f26fa2648b328 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4f7e4478de0f4abeae2f26fa2648b328; no voters:
I20260808 04:04:34.002511 13316 leader_election.cc:288] T 00000000000000000000000000000000 P 4f7e4478de0f4abeae2f26fa2648b328 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:04:34.002617 13323 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4f7e4478de0f4abeae2f26fa2648b328 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:04:34.003253 13323 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4f7e4478de0f4abeae2f26fa2648b328 [term 1 LEADER]: Becoming Leader. State: Replica: 4f7e4478de0f4abeae2f26fa2648b328, State: Running, Role: LEADER
I20260808 04:04:34.003768 13323 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f7e4478de0f4abeae2f26fa2648b328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f7e4478de0f4abeae2f26fa2648b328" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 37771 } }
I20260808 04:04:34.004462 13316 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f7e4478de0f4abeae2f26fa2648b328 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:04:34.006405 13324 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f7e4478de0f4abeae2f26fa2648b328 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4f7e4478de0f4abeae2f26fa2648b328" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f7e4478de0f4abeae2f26fa2648b328" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 37771 } } }
I20260808 04:04:34.006829 13324 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f7e4478de0f4abeae2f26fa2648b328 [sys.catalog]: This master's current role is: LEADER
I20260808 04:04:34.006904 13326 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f7e4478de0f4abeae2f26fa2648b328 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f7e4478de0f4abeae2f26fa2648b328. Latest consensus state: current_term: 1 leader_uuid: "4f7e4478de0f4abeae2f26fa2648b328" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f7e4478de0f4abeae2f26fa2648b328" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 37771 } } }
I20260808 04:04:34.007215 13326 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f7e4478de0f4abeae2f26fa2648b328 [sys.catalog]: This master's current role is: LEADER
I20260808 04:04:34.008673 13339 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:04:34.013558 13339 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:04:34.022071 13339 catalog_manager.cc:1347] Generated new cluster ID: c529e2a1807545bc9eb195321eb86706
I20260808 04:04:34.022166 13339 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:04:34.037230 13339 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:04:34.038427 13339 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:04:34.050818 13339 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4f7e4478de0f4abeae2f26fa2648b328: Generated new TSK 0
I20260808 04:04:34.051589 13339 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.83.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.1.keytab.
Entry for principal kudu/127.5.83.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.1.keytab.
I20260808 04:04:34.114660 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.1:0
--local_ip_for_outbound_sockets=127.5.83.1
--webserver_interface=127.5.83.1
--webserver_port=0
--tserver_master_addrs=127.5.83.62:37771
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.1.keytab
--principal=kudu/127.5.83.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:43547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:04:34.433012 13361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:04:34.433269 13361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:04:34.433400 13361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:04:34.440565 13361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:04:34.440680 13361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:04:34.440838 13361 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.1
I20260808 04:04:34.450420 13361 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:43547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.1.keytab
--principal=kudu/127.5.83.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.83.1
--webserver_port=0
--tserver_master_addrs=127.5.83.62:37771
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.1
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:04:34.451627 13361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:04:34.453014 13361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:04:34.465435 13367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:34.465497 13368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:34.465476 13370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:04:34.466177 13361 server_base.cc:1034] running on GCE node
I20260808 04:04:34.466735 13361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:04:34.467586 13361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:04:34.468786 13361 hybrid_clock.cc:648] HybridClock initialized: now 1786161874468725 us; error 49 us; skew 500 ppm
I20260808 04:04:34.469091 13361 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:04:34 dist-test-slave-1l3l krb5kdc[13198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161874, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:04:34.472149 13361 init.cc:377] Logged in from keytab as kudu/127.5.83.1@KRBTEST.COM (short username kudu)
I20260808 04:04:34.473542 13361 webserver.cc:460] Webserver started at http://127.5.83.1:37077/ using document root <none> and password file <none>
I20260808 04:04:34.474148 13361 fs_manager.cc:359] Metadata directory not provided
I20260808 04:04:34.474267 13361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:04:34.474555 13361 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:04:34.477361 13361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/minicluster-data/ts-0/data/instance:
uuid: "e8051824cb6c4e61ba5d58d6230609e8"
format_stamp: "Formatted at 2026-08-08 04:04:34 on dist-test-slave-1l3l"
server_key: "27a77f6cd2d976344f157edfd45d702a"
server_key_iv: "47c1d429f9f22caadc05c6f54dd27f82"
server_key_version: "encryptionkey@0"
I20260808 04:04:34.478111 13361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/minicluster-data/ts-0/wal/instance:
uuid: "e8051824cb6c4e61ba5d58d6230609e8"
format_stamp: "Formatted at 2026-08-08 04:04:34 on dist-test-slave-1l3l"
server_key: "27a77f6cd2d976344f157edfd45d702a"
server_key_iv: "47c1d429f9f22caadc05c6f54dd27f82"
server_key_version: "encryptionkey@0"
I20260808 04:04:34.482941 13361 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260808 04:04:34.486150 13379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:34.487349 13361 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260808 04:04:34.487504 13361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/minicluster-data/ts-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/minicluster-data/ts-0/wal
uuid: "e8051824cb6c4e61ba5d58d6230609e8"
format_stamp: "Formatted at 2026-08-08 04:04:34 on dist-test-slave-1l3l"
server_key: "27a77f6cd2d976344f157edfd45d702a"
server_key_iv: "47c1d429f9f22caadc05c6f54dd27f82"
server_key_version: "encryptionkey@0"
I20260808 04:04:34.487684 13361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:04:34.502246 13361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:04:34.506201 13361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:04:34.506438 13361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:04:34.507257 13361 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:04:34.508711 13361 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:04:34.508786 13361 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:34.508893 13361 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:04:34.508944 13361 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:34.538609 13361 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.1:33407
I20260808 04:04:34.538653 13543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.1:33407 every 8 connection(s)
I20260808 04:04:34.539796 13361 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
I20260808 04:04:34.544133 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 13361
I20260808 04:04:34.544301 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/minicluster-data/ts-0/wal/instance
I20260808 04:04:34.544912 5452 external_mini_cluster.cc:1428] Setting key 0d8d5546f8f35c1e653f54f5fe775a00
Aug 08 04:04:34 dist-test-slave-1l3l krb5kdc[13198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161874, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.1@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:04:34.559377 13550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:34.542108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.1:34783 (local address 127.5.83.62:37771)
0808 04:04:34.542564 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:34.542571 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:34.543456 (+ 885us) server_negotiation.cc:407] Connection header received
0808 04:04:34.545297 (+ 1841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:34.545302 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:34.545396 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:34.545506 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:34.547303 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:34.548368 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:34.549847 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:34.550418 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:34.550534 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:34.553836 (+ 3302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:34.553842 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:34.553846 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:34.553880 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:34.556573 (+ 2693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:34.557351 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:34.557354 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:34.557356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:34.557480 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:34.557909 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:34.557912 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:34.557913 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:34.558157 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:34.558260 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:34.558966 (+ 706us) server_negotiation.cc:299] Negotiation successful
0808 04:04:34.559128 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":211,"threads_started":1}
I20260808 04:04:34.560658 13554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:34.542587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:37771 (local address 127.5.83.1:34783)
0808 04:04:34.543303 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:34.543338 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:34.545052 (+ 1714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:34.545752 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:34.545760 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:34.546441 (+ 681us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:04:34.547141 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:34.547158 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:34.548532 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:34.548536 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:34.549704 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:34.549716 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:34.550603 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:34.550609 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:34.550805 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:34.551467 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:04:34.551489 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:04:34.553631 (+ 2142us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:04:34.556701 (+ 3070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:34.556709 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:34.556723 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:34.557258 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:34.557574 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:34.557578 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:34.557581 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:34.557817 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:34.558281 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:04:34.558289 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:04:34.558651 (+ 362us) client_negotiation.cc:769] Sending connection context
0808 04:04:34.558932 (+ 281us) client_negotiation.cc:240] Negotiation successful
0808 04:04:34.559382 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":299,"threads_started":1}
I20260808 04:04:34.562166 13545 heartbeater.cc:347] Connected to a master server at 127.5.83.62:37771
I20260808 04:04:34.562431 13545 heartbeater.cc:464] Registering TS with master...
I20260808 04:04:34.563110 13545 heartbeater.cc:511] Master 127.5.83.62:37771 requested a full tablet report, sending...
I20260808 04:04:34.565081 13267 ts_manager.cc:194] Registered new tserver with Master: e8051824cb6c4e61ba5d58d6230609e8 (127.5.83.1:33407)
I20260808 04:04:34.566447 13267 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.1@KRBTEST.COM'} at 127.5.83.1:34783
WARNING: no policy specified for kudu/127.5.83.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.2.keytab.
Entry for principal kudu/127.5.83.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.2.keytab.
I20260808 04:04:34.595546 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.2:0
--local_ip_for_outbound_sockets=127.5.83.2
--webserver_interface=127.5.83.2
--webserver_port=0
--tserver_master_addrs=127.5.83.62:37771
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.2.keytab
--principal=kudu/127.5.83.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:43547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:04:34.908576 13558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:04:34.908833 13558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:04:34.908977 13558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:04:34.916028 13558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:04:34.916142 13558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:04:34.916283 13558 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.2
I20260808 04:04:34.925558 13558 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:43547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.2.keytab
--principal=kudu/127.5.83.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.83.2
--webserver_port=0
--tserver_master_addrs=127.5.83.62:37771
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.2
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:04:34.926743 13558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:04:34.928035 13558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:04:34.940270 13564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:34.940225 13565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:34.941411 13567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:04:34.942085 13558 server_base.cc:1034] running on GCE node
I20260808 04:04:34.942600 13558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:04:34.943434 13558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:04:34.944625 13558 hybrid_clock.cc:648] HybridClock initialized: now 1786161874944584 us; error 33 us; skew 500 ppm
I20260808 04:04:34.944913 13558 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:04:34 dist-test-slave-1l3l krb5kdc[13198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161874, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:04:34.947889 13558 init.cc:377] Logged in from keytab as kudu/127.5.83.2@KRBTEST.COM (short username kudu)
I20260808 04:04:34.949209 13558 webserver.cc:460] Webserver started at http://127.5.83.2:33481/ using document root <none> and password file <none>
I20260808 04:04:34.949811 13558 fs_manager.cc:359] Metadata directory not provided
I20260808 04:04:34.949936 13558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:04:34.950227 13558 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:04:34.953121 13558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/minicluster-data/ts-1/data/instance:
uuid: "5b547e895dd14707968c595db50a4247"
format_stamp: "Formatted at 2026-08-08 04:04:34 on dist-test-slave-1l3l"
server_key: "44d4a12bd2b7647ab0112ce03d698eb6"
server_key_iv: "a2b0ebacb9cf30e55b0b21f1d12ff3fc"
server_key_version: "encryptionkey@0"
I20260808 04:04:34.953878 13558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/minicluster-data/ts-1/wal/instance:
uuid: "5b547e895dd14707968c595db50a4247"
format_stamp: "Formatted at 2026-08-08 04:04:34 on dist-test-slave-1l3l"
server_key: "44d4a12bd2b7647ab0112ce03d698eb6"
server_key_iv: "a2b0ebacb9cf30e55b0b21f1d12ff3fc"
server_key_version: "encryptionkey@0"
I20260808 04:04:34.958683 13558 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260808 04:04:34.961905 13576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:34.963085 13558 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260808 04:04:34.963218 13558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/minicluster-data/ts-1/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/minicluster-data/ts-1/wal
uuid: "5b547e895dd14707968c595db50a4247"
format_stamp: "Formatted at 2026-08-08 04:04:34 on dist-test-slave-1l3l"
server_key: "44d4a12bd2b7647ab0112ce03d698eb6"
server_key_iv: "a2b0ebacb9cf30e55b0b21f1d12ff3fc"
server_key_version: "encryptionkey@0"
I20260808 04:04:34.963392 13558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:04:34.997679 13558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:04:35.001291 13558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:04:35.001564 13558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:04:35.002435 13558 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:04:35.003687 13558 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:04:35.003778 13558 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:35.003899 13558 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:04:35.003961 13558 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:35.037420 13558 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.2:32789
I20260808 04:04:35.037470 13742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.2:32789 every 8 connection(s)
I20260808 04:04:35.038640 13558 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
I20260808 04:04:35.043766 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 13558
I20260808 04:04:35.043939 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/minicluster-data/ts-1/wal/instance
I20260808 04:04:35.044523 5452 external_mini_cluster.cc:1428] Setting key 6efe8b01f89d4e509a3b06ca1743a49c
Aug 08 04:04:35 dist-test-slave-1l3l krb5kdc[13198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161874, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.2@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:04:35.058591 13550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:35.041033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.2:52635 (local address 127.5.83.62:37771)
0808 04:04:35.041203 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:35.041209 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:35.042326 (+ 1117us) server_negotiation.cc:407] Connection header received
0808 04:04:35.044355 (+ 2029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:35.044359 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:35.044437 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:35.044522 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:35.046188 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:35.047026 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:35.049140 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:35.049540 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:35.049633 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:35.053316 (+ 3683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:35.053322 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:35.053325 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:35.053356 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:35.055691 (+ 2335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:35.056395 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:35.056405 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:35.056408 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:35.056536 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:35.056951 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:35.056954 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:35.056956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:35.057201 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:35.057330 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:35.058000 (+ 670us) server_negotiation.cc:299] Negotiation successful
0808 04:04:35.058242 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 04:04:35.059456 13746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:35.041502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:37771 (local address 127.5.83.2:52635)
0808 04:04:35.042184 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:35.042221 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:35.044076 (+ 1855us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:35.044794 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:35.044805 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:35.045424 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:04:35.046023 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:35.046040 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:35.047186 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:35.047191 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:35.049009 (+ 1818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:35.049024 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:35.049745 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:35.049753 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:35.050020 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:35.050672 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:04:35.050698 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:04:35.053114 (+ 2416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:04:35.055846 (+ 2732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:35.055854 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:35.055869 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:35.056270 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:35.056647 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:35.056653 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:35.056656 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:35.056834 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:35.057323 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:04:35.057330 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:04:35.057630 (+ 300us) client_negotiation.cc:769] Sending connection context
0808 04:04:35.057866 (+ 236us) client_negotiation.cc:240] Negotiation successful
0808 04:04:35.058363 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":299,"threads_started":1}
I20260808 04:04:35.060848 13743 heartbeater.cc:347] Connected to a master server at 127.5.83.62:37771
I20260808 04:04:35.061126 13743 heartbeater.cc:464] Registering TS with master...
I20260808 04:04:35.061914 13743 heartbeater.cc:511] Master 127.5.83.62:37771 requested a full tablet report, sending...
I20260808 04:04:35.063308 13267 ts_manager.cc:194] Registered new tserver with Master: 5b547e895dd14707968c595db50a4247 (127.5.83.2:32789)
I20260808 04:04:35.064121 13267 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.2@KRBTEST.COM'} at 127.5.83.2:52635
WARNING: no policy specified for kudu/127.5.83.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.3.keytab.
Entry for principal kudu/127.5.83.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.3.keytab.
I20260808 04:04:35.094125 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.3:0
--local_ip_for_outbound_sockets=127.5.83.3
--webserver_interface=127.5.83.3
--webserver_port=0
--tserver_master_addrs=127.5.83.62:37771
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.3.keytab
--principal=kudu/127.5.83.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:43547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:04:35.409538 13751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:04:35.409783 13751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:04:35.409884 13751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:04:35.416756 13751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:04:35.416847 13751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:04:35.416958 13751 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.3
I20260808 04:04:35.425994 13751 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:43547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.3.keytab
--principal=kudu/127.5.83.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.83.3
--webserver_port=0
--tserver_master_addrs=127.5.83.62:37771
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.3
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:04:35.427127 13751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:04:35.428414 13751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:04:35.440974 13757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:35.441008 13758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:35.441035 13761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:04:35.441830 13751 server_base.cc:1034] running on GCE node
I20260808 04:04:35.442339 13751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:04:35.443214 13751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:04:35.444428 13751 hybrid_clock.cc:648] HybridClock initialized: now 1786161875444355 us; error 66 us; skew 500 ppm
I20260808 04:04:35.444713 13751 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:04:35 dist-test-slave-1l3l krb5kdc[13198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161875, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:04:35.447748 13751 init.cc:377] Logged in from keytab as kudu/127.5.83.3@KRBTEST.COM (short username kudu)
I20260808 04:04:35.449215 13751 webserver.cc:460] Webserver started at http://127.5.83.3:41871/ using document root <none> and password file <none>
I20260808 04:04:35.449859 13751 fs_manager.cc:359] Metadata directory not provided
I20260808 04:04:35.449988 13751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:04:35.450290 13751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:04:35.453346 13751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/minicluster-data/ts-2/data/instance:
uuid: "dab25207b1034f18aa8dc0945a71605c"
format_stamp: "Formatted at 2026-08-08 04:04:35 on dist-test-slave-1l3l"
server_key: "d3b86464740aee170d2f1b382ac21566"
server_key_iv: "e7eae71f684dcd338ec6752b977b8727"
server_key_version: "encryptionkey@0"
I20260808 04:04:35.454213 13751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/minicluster-data/ts-2/wal/instance:
uuid: "dab25207b1034f18aa8dc0945a71605c"
format_stamp: "Formatted at 2026-08-08 04:04:35 on dist-test-slave-1l3l"
server_key: "d3b86464740aee170d2f1b382ac21566"
server_key_iv: "e7eae71f684dcd338ec6752b977b8727"
server_key_version: "encryptionkey@0"
I20260808 04:04:35.459208 13751 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 04:04:35.462507 13769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:35.463762 13751 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:04:35.463976 13751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/minicluster-data/ts-2/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/minicluster-data/ts-2/wal
uuid: "dab25207b1034f18aa8dc0945a71605c"
format_stamp: "Formatted at 2026-08-08 04:04:35 on dist-test-slave-1l3l"
server_key: "d3b86464740aee170d2f1b382ac21566"
server_key_iv: "e7eae71f684dcd338ec6752b977b8727"
server_key_version: "encryptionkey@0"
I20260808 04:04:35.464210 13751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:04:35.494829 13751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:04:35.498657 13751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:04:35.498910 13751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:04:35.499787 13751 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:04:35.501032 13751 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:04:35.501106 13751 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:35.501194 13751 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:04:35.501236 13751 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:35.531880 13751 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.3:46147
I20260808 04:04:35.531917 13927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.3:46147 every 8 connection(s)
I20260808 04:04:35.533115 13751 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
I20260808 04:04:35.542789 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 13751
I20260808 04:04:35.542975 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/minicluster-data/ts-2/wal/instance
I20260808 04:04:35.543411 5452 external_mini_cluster.cc:1428] Setting key f9924e4e5e20c43d2705311200e83f4c
Aug 08 04:04:35 dist-test-slave-1l3l krb5kdc[13198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161875, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.3@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:04:35.551749 13550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:35.535485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.3:44375 (local address 127.5.83.62:37771)
0808 04:04:35.535665 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:35.535671 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:35.536807 (+ 1136us) server_negotiation.cc:407] Connection header received
0808 04:04:35.538601 (+ 1794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:35.538607 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:35.538688 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:35.538777 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:35.540222 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:35.540981 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:35.542847 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:35.543310 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:35.543438 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:35.546884 (+ 3446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:35.546889 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:35.546892 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:35.546934 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:35.549290 (+ 2356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:35.549938 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:35.549942 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:35.549944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:35.550067 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:35.550403 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:35.550406 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:35.550408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:35.550662 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:35.550768 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:35.551337 (+ 569us) server_negotiation.cc:299] Negotiation successful
0808 04:04:35.551492 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 04:04:35.552614 13930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:35.535988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:37771 (local address 127.5.83.3:44375)
0808 04:04:35.536662 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:35.536699 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:35.538346 (+ 1647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:35.539011 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:35.539018 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:35.539482 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:04:35.540056 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:35.540075 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:35.541131 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:35.541135 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:35.542666 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:35.542679 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:35.543538 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:35.543546 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:35.543801 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:35.544417 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:04:35.544439 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:04:35.546701 (+ 2262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:04:35.549420 (+ 2719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:35.549426 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:35.549436 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:35.549835 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:35.550145 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:35.550148 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:35.550150 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:35.550317 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:35.550747 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:04:35.550752 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:04:35.551049 (+ 297us) client_negotiation.cc:769] Sending connection context
0808 04:04:35.551251 (+ 202us) client_negotiation.cc:240] Negotiation successful
0808 04:04:35.551588 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":284,"threads_started":1}
I20260808 04:04:35.553802 13928 heartbeater.cc:347] Connected to a master server at 127.5.83.62:37771
I20260808 04:04:35.554071 13928 heartbeater.cc:464] Registering TS with master...
I20260808 04:04:35.554735 13928 heartbeater.cc:511] Master 127.5.83.62:37771 requested a full tablet report, sending...
I20260808 04:04:35.556136 13267 ts_manager.cc:194] Registered new tserver with Master: dab25207b1034f18aa8dc0945a71605c (127.5.83.3:46147)
I20260808 04:04:35.557016 13267 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.3@KRBTEST.COM'} at 127.5.83.3:44375
I20260808 04:04:35.558564 5452 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:04:35.569123 13545 heartbeater.cc:503] Master 127.5.83.62:37771 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/krb5kdc/test-admin.keytab.
Aug 08 04:04:35 dist-test-slave-1l3l krb5kdc[13198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:04:35.588336 5452 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260808 04:04:35.593379 13941 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:04:35.590752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34644 (local address 127.5.83.1:33407)
0808 04:04:35.591402 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:35.591409 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:35.591479 (+ 70us) server_negotiation.cc:407] Connection header received
0808 04:04:35.591623 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:35.591629 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:35.591878 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:35.592019 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:35.592903 (+ 884us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34644: BlockingRecv error: recv got EOF from 127.0.0.1:34644 (error 108)
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":349,"threads_started":1}
W20260808 04:04:35.593357 5452 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/krb5kdc/krb5cc' of type FILE
Aug 08 04:04:35 dist-test-slave-1l3l krb5kdc[13198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:04:35.594988 5452 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 08 04:04:35 dist-test-slave-1l3l krb5kdc[13198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:04:35.596489 5452 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 08 04:04:35 dist-test-slave-1l3l krb5kdc[13198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.1@KRBTEST.COM
I20260808 04:04:35.610965 13941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:35.598055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34658 (local address 127.5.83.1:33407)
0808 04:04:35.598238 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:35.598244 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:35.598458 (+ 214us) server_negotiation.cc:407] Connection header received
0808 04:04:35.598620 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:35.598624 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:35.598682 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:35.598762 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:35.599764 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:35.600717 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:35.601959 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:35.602497 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:35.602610 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:35.606067 (+ 3457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:35.606078 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:35.606092 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:35.606131 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:35.608715 (+ 2584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:35.609314 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:35.609320 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:35.609323 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:35.609478 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:35.609766 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:35.609768 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:35.609770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:35.610086 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:35.610305 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:35.610522 (+ 217us) server_negotiation.cc:299] Negotiation successful
0808 04:04:35.610719 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260808 04:04:35.616070 13941 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:04:35.614696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34672 (local address 127.5.83.1:33407)
0808 04:04:35.614888 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:35.614894 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:35.615022 (+ 128us) server_negotiation.cc:407] Connection header received
0808 04:04:35.615136 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:35.615139 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:35.615202 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:35.615273 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:35.615855 (+ 582us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34672: BlockingRecv error: recv got EOF from 127.0.0.1:34672 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260808 04:04:35.616273 5452 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/krb5kdc/krb5cc' of type FILE
Aug 08 04:04:35 dist-test-slave-1l3l krb5kdc[13198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:04:35.617630 5452 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 08 04:04:35 dist-test-slave-1l3l krb5kdc[13198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:04:35.618779 5452 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 08 04:04:35 dist-test-slave-1l3l krb5kdc[13198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.1@KRBTEST.COM
I20260808 04:04:35.631294 13941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:35.619957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34682 (local address 127.5.83.1:33407)
0808 04:04:35.620085 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:35.620089 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:35.620437 (+ 348us) server_negotiation.cc:407] Connection header received
0808 04:04:35.620543 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:35.620546 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:35.620594 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:35.620656 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:35.621487 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:35.622173 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:35.623271 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:35.623693 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:35.623813 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:35.626969 (+ 3156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:35.626974 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:35.626978 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:35.627007 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:35.629165 (+ 2158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:35.629738 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:35.629741 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:35.629743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:35.629853 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:35.630112 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:35.630115 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:35.630117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:35.630366 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:35.630564 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:35.630906 (+ 342us) server_negotiation.cc:299] Negotiation successful
0808 04:04:35.631058 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260808 04:04:35.635248 13941 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:04:35.633696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34690 (local address 127.5.83.1:33407)
0808 04:04:35.633892 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:35.633897 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:35.634042 (+ 145us) server_negotiation.cc:407] Connection header received
0808 04:04:35.634148 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:35.634151 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:35.634214 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:35.634284 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:35.635011 (+ 727us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34690: BlockingRecv error: recv got EOF from 127.0.0.1:34690 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260808 04:04:35.636149 5452 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161826605655-5452-0/krb5kdc/krb5cc' of type FILE
Aug 08 04:04:35 dist-test-slave-1l3l krb5kdc[13198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:04:35.637593 5452 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 08 04:04:35 dist-test-slave-1l3l krb5kdc[13198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:04:35.638823 5452 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 08 04:04:35 dist-test-slave-1l3l krb5kdc[13198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.1@KRBTEST.COM
I20260808 04:04:35.651928 13941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:35.640011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34700 (local address 127.5.83.1:33407)
0808 04:04:35.640200 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:35.640205 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:35.640438 (+ 233us) server_negotiation.cc:407] Connection header received
0808 04:04:35.640514 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:35.640518 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:35.640572 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:35.640638 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:35.641513 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:35.642233 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:35.643409 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:35.643934 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:35.644030 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:35.647100 (+ 3070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:35.647104 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:35.647108 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:35.647137 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:35.649680 (+ 2543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:35.650203 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:35.650206 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:35.650209 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:35.650409 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:35.650671 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:35.650674 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:35.650675 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:35.650929 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:35.651146 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:35.651402 (+ 256us) server_negotiation.cc:299] Negotiation successful
0808 04:04:35.651602 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 04:04:35.653378 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 13361
I20260808 04:04:35.663887 13941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:35.654026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34716 (local address 127.5.83.1:33407)
0808 04:04:35.654201 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:35.654207 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:35.654472 (+ 265us) server_negotiation.cc:407] Connection header received
0808 04:04:35.654559 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:35.654562 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:35.654611 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:35.654680 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:35.655488 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:35.656179 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:35.657359 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:35.657804 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:35.657896 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:35.659646 (+ 1750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:35.659650 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:35.659653 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:35.659676 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:35.661854 (+ 2178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:35.662376 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:35.662402 (+ 26us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:35.662404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:35.662525 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:35.662792 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:35.662795 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:35.662797 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:35.663002 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:35.663186 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:35.663378 (+ 192us) server_negotiation.cc:299] Negotiation successful
0808 04:04:35.663574 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 04:04:35.664543 13538 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:04:35.746088 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 13361
I20260808 04:04:35.767868 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 13558
Aug 08 04:04:35 dist-test-slave-1l3l krb5kdc[13198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.2@KRBTEST.COM
I20260808 04:04:35.782287 13957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:35.768801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55896 (local address 127.5.83.2:32789)
0808 04:04:35.769326 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:35.769333 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:35.769358 (+ 25us) server_negotiation.cc:407] Connection header received
0808 04:04:35.769446 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:35.769453 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:35.769664 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:35.769823 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:35.770784 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:35.771821 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:35.772958 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:35.773487 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:35.773588 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:35.777173 (+ 3585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:35.777185 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:35.777199 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:35.777231 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:35.779968 (+ 2737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:35.780585 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:35.780592 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:35.780594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:35.780705 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:35.781007 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:35.781010 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:35.781012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:35.781313 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:35.781533 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:35.781781 (+ 248us) server_negotiation.cc:299] Negotiation successful
0808 04:04:35.781980 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":257,"threads_started":1}
I20260808 04:04:35.783823 13736 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:04:35.864465 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 13558
I20260808 04:04:35.881340 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 13751
Aug 08 04:04:35 dist-test-slave-1l3l krb5kdc[13198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.3@KRBTEST.COM
I20260808 04:04:35.895860 13959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:35.882426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41886 (local address 127.5.83.3:46147)
0808 04:04:35.883093 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:35.883099 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:35.883128 (+ 29us) server_negotiation.cc:407] Connection header received
0808 04:04:35.883190 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:35.883196 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:35.883400 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:35.883524 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:35.884549 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:35.885550 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:35.886670 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:35.887201 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:35.887292 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:35.891015 (+ 3723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:35.891026 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:35.891038 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:35.891070 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:35.893627 (+ 2557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:35.894179 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:35.894185 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:35.894187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:35.894296 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:35.894564 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:35.894567 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:35.894569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:35.894875 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:35.895114 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:35.895315 (+ 201us) server_negotiation.cc:299] Negotiation successful
0808 04:04:35.895528 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":365,"threads_started":1}
I20260808 04:04:35.897138 13921 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:04:35.977802 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 13751
I20260808 04:04:35.997225 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 13216
I20260808 04:04:35.998361 13307 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:04:36.102612 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 13216
2026-08-08T04:04:36Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3903 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:04:36 dist-test-slave-1l3l krb5kdc[13966](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:04:36 dist-test-slave-1l3l krb5kdc[13966](info): set up 2 sockets
Aug 08 04:04:36 dist-test-slave-1l3l krb5kdc[13966](info): commencing operation
krb5kdc: starting...
W20260808 04:04:37.297973 5452 mini_kdc.cc:121] Time spent starting KDC: real 1.150s user 0.005s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/krb5kdc/test-user.keytab.
Aug 08 04:04:37 dist-test-slave-1l3l krb5kdc[13966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:04:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:04:37Z Disabled control of system clock
WARNING: no policy specified for oryx/127.5.83.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.83.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.83.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/krb5kdc/oryx_127.5.83.62.keytab.
Entry for principal oryx/127.5.83.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/krb5kdc/oryx_127.5.83.62.keytab.
I20260808 04:04:37.506868 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:39395
--webserver_interface=127.5.83.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/krb5kdc/oryx_127.5.83.62.keytab
--principal=oryx/127.5.83.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:46157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.62:39395
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:04:37.814010 13985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:04:37.814257 13985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:04:37.814368 13985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:04:37.821151 13985 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:04:37.821242 13985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:04:37.821314 13985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:04:37.821393 13985 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:04:37.821453 13985 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:04:37.830696 13985 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:46157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.62:39395
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/krb5kdc/oryx_127.5.83.62.keytab
--principal=oryx/127.5.83.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:39395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:04:37.831876 13985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:04:37.833122 13985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:04:37.843997 13991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:37.843973 13992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:37.844256 13996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:04:37.844913 13985 server_base.cc:1034] running on GCE node
I20260808 04:04:37.845633 13985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:04:37.846904 13985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:04:37.848096 13985 hybrid_clock.cc:648] HybridClock initialized: now 1786161877848050 us; error 32 us; skew 500 ppm
I20260808 04:04:37.848393 13985 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:04:37 dist-test-slave-1l3l krb5kdc[13966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161877, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.83.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:04:37.851238 13985 init.cc:377] Logged in from keytab as oryx/127.5.83.62@KRBTEST.COM (short username oryx)
I20260808 04:04:37.852656 13985 webserver.cc:460] Webserver started at http://127.5.83.62:46413/ using document root <none> and password file <none>
I20260808 04:04:37.853304 13985 fs_manager.cc:359] Metadata directory not provided
I20260808 04:04:37.853417 13985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:04:37.853667 13985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:04:37.856489 13985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/master-0/data/instance:
uuid: "b254403e06ed4a0fbad7433dbbb9f425"
format_stamp: "Formatted at 2026-08-08 04:04:37 on dist-test-slave-1l3l"
server_key: "a198ff59c8878a218fa754b8497616ec"
server_key_iv: "865884af9750a20483ed476992c2ca17"
server_key_version: "encryptionkey@0"
I20260808 04:04:37.857187 13985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/master-0/wal/instance:
uuid: "b254403e06ed4a0fbad7433dbbb9f425"
format_stamp: "Formatted at 2026-08-08 04:04:37 on dist-test-slave-1l3l"
server_key: "a198ff59c8878a218fa754b8497616ec"
server_key_iv: "865884af9750a20483ed476992c2ca17"
server_key_version: "encryptionkey@0"
I20260808 04:04:37.861843 13985 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260808 04:04:37.865020 14010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:37.866228 13985 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 04:04:37.866377 13985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/master-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/master-0/wal
uuid: "b254403e06ed4a0fbad7433dbbb9f425"
format_stamp: "Formatted at 2026-08-08 04:04:37 on dist-test-slave-1l3l"
server_key: "a198ff59c8878a218fa754b8497616ec"
server_key_iv: "865884af9750a20483ed476992c2ca17"
server_key_version: "encryptionkey@0"
I20260808 04:04:37.866567 13985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:04:37.885437 13985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:04:37.889472 13985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:04:37.889721 13985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:04:37.907126 13985 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.62:39395
I20260808 04:04:37.907159 14086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.62:39395 every 8 connection(s)
I20260808 04:04:37.908438 13985 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
I20260808 04:04:37.911895 14087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:04:37.913942 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 13985
I20260808 04:04:37.914148 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/master-0/wal/instance
I20260808 04:04:37.914942 5452 external_mini_cluster.cc:1428] Setting key 8bb2d573e2ada00ba58d7e92635c3cc6
I20260808 04:04:37.923506 14087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b254403e06ed4a0fbad7433dbbb9f425: Bootstrap starting.
I20260808 04:04:37.926942 14087 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b254403e06ed4a0fbad7433dbbb9f425: Neither blocks nor log segments found. Creating new log.
I20260808 04:04:37.928167 14087 log.cc:824] T 00000000000000000000000000000000 P b254403e06ed4a0fbad7433dbbb9f425: Log is configured to *not* fsync() on all Append() calls
Aug 08 04:04:37 dist-test-slave-1l3l krb5kdc[13966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM
I20260808 04:04:37.931037 14087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b254403e06ed4a0fbad7433dbbb9f425: No bootstrap required, opened a new log
I20260808 04:04:37.936903 14091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:37.916992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43172 (local address 127.5.83.62:39395)
0808 04:04:37.917736 (+ 744us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:37.917748 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:37.917786 (+ 38us) server_negotiation.cc:407] Connection header received
0808 04:04:37.919344 (+ 1558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:37.919364 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:37.919732 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:37.920059 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:37.921692 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:37.922925 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:37.924602 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:37.925250 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:37.925382 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:37.930156 (+ 4774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:37.930164 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:37.930177 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:37.930209 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:37.933121 (+ 2912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:37.933887 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:37.933892 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:37.933897 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:37.934048 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:37.934408 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:37.934412 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:37.934414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:37.934905 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:37.935129 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:37.935550 (+ 421us) server_negotiation.cc:299] Negotiation successful
0808 04:04:37.935930 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":319,"threads_started":1}
I20260808 04:04:37.939150 14087 raft_consensus.cc:348] T 00000000000000000000000000000000 P b254403e06ed4a0fbad7433dbbb9f425 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b254403e06ed4a0fbad7433dbbb9f425" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 39395 } }
I20260808 04:04:37.939483 14087 raft_consensus.cc:374] T 00000000000000000000000000000000 P b254403e06ed4a0fbad7433dbbb9f425 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:04:37.939574 14087 raft_consensus.cc:729] T 00000000000000000000000000000000 P b254403e06ed4a0fbad7433dbbb9f425 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b254403e06ed4a0fbad7433dbbb9f425, State: Initialized, Role: FOLLOWER
I20260808 04:04:37.940088 14087 consensus_queue.cc:260] T 00000000000000000000000000000000 P b254403e06ed4a0fbad7433dbbb9f425 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b254403e06ed4a0fbad7433dbbb9f425" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 39395 } }
I20260808 04:04:37.940291 14087 raft_consensus.cc:388] T 00000000000000000000000000000000 P b254403e06ed4a0fbad7433dbbb9f425 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:04:37.940411 14087 raft_consensus.cc:482] T 00000000000000000000000000000000 P b254403e06ed4a0fbad7433dbbb9f425 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:04:37.940555 14087 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b254403e06ed4a0fbad7433dbbb9f425 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:04:37.942534 14087 raft_consensus.cc:504] T 00000000000000000000000000000000 P b254403e06ed4a0fbad7433dbbb9f425 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b254403e06ed4a0fbad7433dbbb9f425" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 39395 } }
I20260808 04:04:37.942977 14087 leader_election.cc:302] T 00000000000000000000000000000000 P b254403e06ed4a0fbad7433dbbb9f425 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b254403e06ed4a0fbad7433dbbb9f425; no voters:
I20260808 04:04:37.943485 14087 leader_election.cc:288] T 00000000000000000000000000000000 P b254403e06ed4a0fbad7433dbbb9f425 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:04:37.943586 14094 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b254403e06ed4a0fbad7433dbbb9f425 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:04:37.944245 14094 raft_consensus.cc:686] T 00000000000000000000000000000000 P b254403e06ed4a0fbad7433dbbb9f425 [term 1 LEADER]: Becoming Leader. State: Replica: b254403e06ed4a0fbad7433dbbb9f425, State: Running, Role: LEADER
I20260808 04:04:37.944748 14094 consensus_queue.cc:237] T 00000000000000000000000000000000 P b254403e06ed4a0fbad7433dbbb9f425 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b254403e06ed4a0fbad7433dbbb9f425" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 39395 } }
I20260808 04:04:37.945422 14087 sys_catalog.cc:564] T 00000000000000000000000000000000 P b254403e06ed4a0fbad7433dbbb9f425 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:04:37.947489 14096 sys_catalog.cc:455] T 00000000000000000000000000000000 P b254403e06ed4a0fbad7433dbbb9f425 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b254403e06ed4a0fbad7433dbbb9f425. Latest consensus state: current_term: 1 leader_uuid: "b254403e06ed4a0fbad7433dbbb9f425" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b254403e06ed4a0fbad7433dbbb9f425" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 39395 } } }
I20260808 04:04:37.947939 14096 sys_catalog.cc:458] T 00000000000000000000000000000000 P b254403e06ed4a0fbad7433dbbb9f425 [sys.catalog]: This master's current role is: LEADER
I20260808 04:04:37.947834 14095 sys_catalog.cc:455] T 00000000000000000000000000000000 P b254403e06ed4a0fbad7433dbbb9f425 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b254403e06ed4a0fbad7433dbbb9f425" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b254403e06ed4a0fbad7433dbbb9f425" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 39395 } } }
I20260808 04:04:37.948155 14095 sys_catalog.cc:458] T 00000000000000000000000000000000 P b254403e06ed4a0fbad7433dbbb9f425 [sys.catalog]: This master's current role is: LEADER
I20260808 04:04:37.949854 14105 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:04:37.954402 14105 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:04:37.961933 14105 catalog_manager.cc:1347] Generated new cluster ID: a15e7f4b3b484d4dbf31a2eea6d58df7
I20260808 04:04:37.962046 14105 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:04:37.969259 14105 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:04:37.970530 14105 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:04:38.000588 14105 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b254403e06ed4a0fbad7433dbbb9f425: Generated new TSK 0
I20260808 04:04:38.001607 14105 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.5.83.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.83.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.83.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/krb5kdc/oryx_127.5.83.1.keytab.
Entry for principal oryx/127.5.83.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/krb5kdc/oryx_127.5.83.1.keytab.
I20260808 04:04:38.056890 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.1:0
--local_ip_for_outbound_sockets=127.5.83.1
--webserver_interface=127.5.83.1
--webserver_port=0
--tserver_master_addrs=127.5.83.62:39395
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/krb5kdc/oryx_127.5.83.1.keytab
--principal=oryx/127.5.83.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:46157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:04:38.375900 14121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:04:38.376160 14121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:04:38.376286 14121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:04:38.383283 14121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:04:38.383386 14121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:04:38.383528 14121 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.1
I20260808 04:04:38.393808 14121 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:46157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/krb5kdc/oryx_127.5.83.1.keytab
--principal=oryx/127.5.83.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.83.1
--webserver_port=0
--tserver_master_addrs=127.5.83.62:39395
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.1
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:04:38.395079 14121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:04:38.396414 14121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:04:38.408839 14127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:38.408771 14131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:38.409225 14128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:04:38.409718 14121 server_base.cc:1034] running on GCE node
I20260808 04:04:38.410285 14121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:04:38.411150 14121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:04:38.412349 14121 hybrid_clock.cc:648] HybridClock initialized: now 1786161878412298 us; error 31 us; skew 500 ppm
I20260808 04:04:38.412642 14121 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:04:38 dist-test-slave-1l3l krb5kdc[13966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161878, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.83.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:04:38.415738 14121 init.cc:377] Logged in from keytab as oryx/127.5.83.1@KRBTEST.COM (short username oryx)
I20260808 04:04:38.417059 14121 webserver.cc:460] Webserver started at http://127.5.83.1:38921/ using document root <none> and password file <none>
I20260808 04:04:38.417659 14121 fs_manager.cc:359] Metadata directory not provided
I20260808 04:04:38.417759 14121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:04:38.418023 14121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:04:38.420773 14121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-0/data/instance:
uuid: "82a19f4b641448c384fdc9198ab1143e"
format_stamp: "Formatted at 2026-08-08 04:04:38 on dist-test-slave-1l3l"
server_key: "ebba60576942b7c95247b2228dc30bbe"
server_key_iv: "f58da4ec03823b6b87d392522e07330a"
server_key_version: "encryptionkey@0"
I20260808 04:04:38.421564 14121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-0/wal/instance:
uuid: "82a19f4b641448c384fdc9198ab1143e"
format_stamp: "Formatted at 2026-08-08 04:04:38 on dist-test-slave-1l3l"
server_key: "ebba60576942b7c95247b2228dc30bbe"
server_key_iv: "f58da4ec03823b6b87d392522e07330a"
server_key_version: "encryptionkey@0"
I20260808 04:04:38.426611 14121 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 04:04:38.429848 14138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:38.431092 14121 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260808 04:04:38.431246 14121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-0/wal
uuid: "82a19f4b641448c384fdc9198ab1143e"
format_stamp: "Formatted at 2026-08-08 04:04:38 on dist-test-slave-1l3l"
server_key: "ebba60576942b7c95247b2228dc30bbe"
server_key_iv: "f58da4ec03823b6b87d392522e07330a"
server_key_version: "encryptionkey@0"
I20260808 04:04:38.431435 14121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:04:38.464267 14121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:04:38.468456 14121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:04:38.468719 14121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:04:38.469687 14121 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:04:38.471136 14121 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:04:38.471210 14121 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:38.471341 14121 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:04:38.471397 14121 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:38.502363 14121 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.1:45257
I20260808 04:04:38.502445 14300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.1:45257 every 8 connection(s)
I20260808 04:04:38.503566 14121 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
I20260808 04:04:38.506412 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 14121
I20260808 04:04:38.506603 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-0/wal/instance
I20260808 04:04:38.507231 5452 external_mini_cluster.cc:1428] Setting key c1904a7d43689de3786d9808a7e92194
Aug 08 04:04:38 dist-test-slave-1l3l krb5kdc[13966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161878, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.83.1@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM
I20260808 04:04:38.523193 14304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:38.505840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.1:34771 (local address 127.5.83.62:39395)
0808 04:04:38.506282 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:38.506288 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:38.507317 (+ 1029us) server_negotiation.cc:407] Connection header received
0808 04:04:38.509205 (+ 1888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:38.509210 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:38.509288 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:38.509407 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:38.510762 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:38.511771 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:38.513262 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:38.513785 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:38.513915 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:38.517429 (+ 3514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:38.517434 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:38.517438 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:38.517469 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:38.520349 (+ 2880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:38.521008 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:38.521013 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:38.521017 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:38.521154 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:38.521545 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:38.521550 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:38.521554 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:38.521855 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:38.522009 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:38.522596 (+ 587us) server_negotiation.cc:299] Negotiation successful
0808 04:04:38.522835 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":213,"threads_started":1}
I20260808 04:04:38.523947 14305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:38.506399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:39395 (local address 127.5.83.1:34771)
0808 04:04:38.507166 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:38.507212 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:38.508952 (+ 1740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:38.509593 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:38.509601 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:38.510063 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:04:38.510593 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:38.510610 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:38.511927 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:38.511931 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:38.513120 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:38.513132 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:38.513987 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:38.513992 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:38.514185 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:38.514806 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:04:38.514832 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:04:38.517223 (+ 2391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:04:38.520486 (+ 3263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:38.520492 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:38.520503 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:38.520898 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:38.521250 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:38.521253 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:38.521256 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:38.521439 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:38.521963 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:04:38.521968 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:04:38.522239 (+ 271us) client_negotiation.cc:769] Sending connection context
0808 04:04:38.522464 (+ 225us) client_negotiation.cc:240] Negotiation successful
0808 04:04:38.522801 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":341,"threads_started":1}
I20260808 04:04:38.525533 14301 heartbeater.cc:347] Connected to a master server at 127.5.83.62:39395
I20260808 04:04:38.525817 14301 heartbeater.cc:464] Registering TS with master...
I20260808 04:04:38.526489 14301 heartbeater.cc:511] Master 127.5.83.62:39395 requested a full tablet report, sending...
I20260808 04:04:38.528769 14029 ts_manager.cc:194] Registered new tserver with Master: 82a19f4b641448c384fdc9198ab1143e (127.5.83.1:45257)
I20260808 04:04:38.530754 14029 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.83.1@KRBTEST.COM'} at 127.5.83.1:34771
WARNING: no policy specified for oryx/127.5.83.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.83.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.83.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/krb5kdc/oryx_127.5.83.2.keytab.
Entry for principal oryx/127.5.83.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/krb5kdc/oryx_127.5.83.2.keytab.
I20260808 04:04:38.559397 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.2:0
--local_ip_for_outbound_sockets=127.5.83.2
--webserver_interface=127.5.83.2
--webserver_port=0
--tserver_master_addrs=127.5.83.62:39395
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/krb5kdc/oryx_127.5.83.2.keytab
--principal=oryx/127.5.83.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:46157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:04:38.871153 14308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:04:38.871413 14308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:04:38.871546 14308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:04:38.878873 14308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:04:38.879002 14308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:04:38.879164 14308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.2
I20260808 04:04:38.888777 14308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:46157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/krb5kdc/oryx_127.5.83.2.keytab
--principal=oryx/127.5.83.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.83.2
--webserver_port=0
--tserver_master_addrs=127.5.83.62:39395
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.2
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:04:38.889968 14308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:04:38.891218 14308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:04:38.903388 14314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:38.903450 14315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:38.904052 14318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:04:38.904098 14308 server_base.cc:1034] running on GCE node
I20260808 04:04:38.904731 14308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:04:38.905599 14308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:04:38.906795 14308 hybrid_clock.cc:648] HybridClock initialized: now 1786161878906745 us; error 39 us; skew 500 ppm
I20260808 04:04:38.907083 14308 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:04:38 dist-test-slave-1l3l krb5kdc[13966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161878, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.83.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:04:38.910105 14308 init.cc:377] Logged in from keytab as oryx/127.5.83.2@KRBTEST.COM (short username oryx)
I20260808 04:04:38.911535 14308 webserver.cc:460] Webserver started at http://127.5.83.2:41269/ using document root <none> and password file <none>
I20260808 04:04:38.912202 14308 fs_manager.cc:359] Metadata directory not provided
I20260808 04:04:38.912328 14308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:04:38.912616 14308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:04:38.915845 14308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-1/data/instance:
uuid: "861039c94e114a4c9afc2caeb3282448"
format_stamp: "Formatted at 2026-08-08 04:04:38 on dist-test-slave-1l3l"
server_key: "7cfc0a58299f7111a152a93b288496f7"
server_key_iv: "d12fc8583b9c11e6684cbfe1f7346a22"
server_key_version: "encryptionkey@0"
I20260808 04:04:38.916636 14308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-1/wal/instance:
uuid: "861039c94e114a4c9afc2caeb3282448"
format_stamp: "Formatted at 2026-08-08 04:04:38 on dist-test-slave-1l3l"
server_key: "7cfc0a58299f7111a152a93b288496f7"
server_key_iv: "d12fc8583b9c11e6684cbfe1f7346a22"
server_key_version: "encryptionkey@0"
I20260808 04:04:38.921600 14308 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 04:04:38.924795 14329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:38.926004 14308 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20260808 04:04:38.926148 14308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-1/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-1/wal
uuid: "861039c94e114a4c9afc2caeb3282448"
format_stamp: "Formatted at 2026-08-08 04:04:38 on dist-test-slave-1l3l"
server_key: "7cfc0a58299f7111a152a93b288496f7"
server_key_iv: "d12fc8583b9c11e6684cbfe1f7346a22"
server_key_version: "encryptionkey@0"
I20260808 04:04:38.926323 14308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:04:38.945720 14308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:04:38.949987 14308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:04:38.950237 14308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:04:38.951116 14308 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:04:38.952455 14308 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:04:38.952529 14308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:38.952623 14308 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:04:38.952667 14308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:38.982761 14308 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.2:46681
I20260808 04:04:38.982807 14503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.2:46681 every 8 connection(s)
I20260808 04:04:38.984000 14308 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
I20260808 04:04:38.988397 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 14308
I20260808 04:04:38.988565 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-1/wal/instance
I20260808 04:04:38.989012 5452 external_mini_cluster.cc:1428] Setting key 56d6207203b55b3b8b78831102aebcdd
Aug 08 04:04:38 dist-test-slave-1l3l krb5kdc[13966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161878, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.83.2@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM
I20260808 04:04:39.002569 14304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:38.986497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.2:50131 (local address 127.5.83.62:39395)
0808 04:04:38.986676 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:38.986681 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:38.987814 (+ 1133us) server_negotiation.cc:407] Connection header received
0808 04:04:38.989625 (+ 1811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:38.989631 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:38.989710 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:38.989798 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:38.991193 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:38.991932 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:38.993433 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:38.993831 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:38.993951 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:38.997316 (+ 3365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:38.997321 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:38.997325 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:38.997356 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:38.999655 (+ 2299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:39.000391 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:39.000395 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:39.000397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:39.000538 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:39.000987 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:39.000991 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:39.000993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:39.001247 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:39.001387 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:39.002110 (+ 723us) server_negotiation.cc:299] Negotiation successful
0808 04:04:39.002277 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 04:04:39.003784 14507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:38.986915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:39395 (local address 127.5.83.2:50131)
0808 04:04:38.987634 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:38.987677 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:38.989367 (+ 1690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:38.990026 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:38.990033 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:38.990503 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:04:38.991027 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:38.991044 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:38.992085 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:38.992089 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:38.993290 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:38.993302 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:38.994043 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:38.994051 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:38.994257 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:38.994874 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:04:38.994902 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:04:38.997114 (+ 2212us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:04:38.999794 (+ 2680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:38.999803 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:38.999817 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:39.000263 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:39.000642 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:39.000647 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:39.000650 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:39.000867 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:39.001384 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:04:39.001392 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:04:39.001757 (+ 365us) client_negotiation.cc:769] Sending connection context
0808 04:04:39.002038 (+ 281us) client_negotiation.cc:240] Negotiation successful
0808 04:04:39.002535 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":290,"threads_started":1}
I20260808 04:04:39.005216 14504 heartbeater.cc:347] Connected to a master server at 127.5.83.62:39395
I20260808 04:04:39.005540 14504 heartbeater.cc:464] Registering TS with master...
I20260808 04:04:39.006335 14504 heartbeater.cc:511] Master 127.5.83.62:39395 requested a full tablet report, sending...
I20260808 04:04:39.007877 14029 ts_manager.cc:194] Registered new tserver with Master: 861039c94e114a4c9afc2caeb3282448 (127.5.83.2:46681)
I20260808 04:04:39.008658 14029 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.83.2@KRBTEST.COM'} at 127.5.83.2:50131
WARNING: no policy specified for oryx/127.5.83.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.83.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.83.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/krb5kdc/oryx_127.5.83.3.keytab.
Entry for principal oryx/127.5.83.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/krb5kdc/oryx_127.5.83.3.keytab.
I20260808 04:04:39.040875 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.3:0
--local_ip_for_outbound_sockets=127.5.83.3
--webserver_interface=127.5.83.3
--webserver_port=0
--tserver_master_addrs=127.5.83.62:39395
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/krb5kdc/oryx_127.5.83.3.keytab
--principal=oryx/127.5.83.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:46157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:04:39.352146 14518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:04:39.352401 14518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:04:39.352542 14518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:04:39.359494 14518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:04:39.359589 14518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:04:39.359745 14518 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.3
I20260808 04:04:39.368811 14518 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:46157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/krb5kdc/oryx_127.5.83.3.keytab
--principal=oryx/127.5.83.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.83.3
--webserver_port=0
--tserver_master_addrs=127.5.83.62:39395
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.3
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:04:39.370025 14518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:04:39.371300 14518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:04:39.383658 14524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:39.383607 14525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:39.383635 14530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:04:39.384655 14518 server_base.cc:1034] running on GCE node
I20260808 04:04:39.385182 14518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:04:39.386021 14518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:04:39.387215 14518 hybrid_clock.cc:648] HybridClock initialized: now 1786161879387165 us; error 43 us; skew 500 ppm
I20260808 04:04:39.387498 14518 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:04:39 dist-test-slave-1l3l krb5kdc[13966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161879, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.83.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:04:39.390372 14518 init.cc:377] Logged in from keytab as oryx/127.5.83.3@KRBTEST.COM (short username oryx)
I20260808 04:04:39.391749 14518 webserver.cc:460] Webserver started at http://127.5.83.3:39401/ using document root <none> and password file <none>
I20260808 04:04:39.392375 14518 fs_manager.cc:359] Metadata directory not provided
I20260808 04:04:39.392498 14518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:04:39.392781 14518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:04:39.396293 14518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-2/data/instance:
uuid: "4e9480f289294032b1d7d1018594001c"
format_stamp: "Formatted at 2026-08-08 04:04:39 on dist-test-slave-1l3l"
server_key: "90885313ee48aa73315ec34be597f735"
server_key_iv: "2246ad0aa5a18d9e45cd9c695b6e12fa"
server_key_version: "encryptionkey@0"
I20260808 04:04:39.397038 14518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-2/wal/instance:
uuid: "4e9480f289294032b1d7d1018594001c"
format_stamp: "Formatted at 2026-08-08 04:04:39 on dist-test-slave-1l3l"
server_key: "90885313ee48aa73315ec34be597f735"
server_key_iv: "2246ad0aa5a18d9e45cd9c695b6e12fa"
server_key_version: "encryptionkey@0"
I20260808 04:04:39.402056 14518 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260808 04:04:39.405224 14539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:39.406440 14518 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 04:04:39.406590 14518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-2/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-2/wal
uuid: "4e9480f289294032b1d7d1018594001c"
format_stamp: "Formatted at 2026-08-08 04:04:39 on dist-test-slave-1l3l"
server_key: "90885313ee48aa73315ec34be597f735"
server_key_iv: "2246ad0aa5a18d9e45cd9c695b6e12fa"
server_key_version: "encryptionkey@0"
I20260808 04:04:39.406765 14518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:04:39.419210 14518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:04:39.422694 14518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:04:39.422940 14518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:04:39.423820 14518 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:04:39.425050 14518 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:04:39.425122 14518 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:39.425226 14518 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:04:39.425275 14518 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:39.455471 14518 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.3:46513
I20260808 04:04:39.455554 14706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.3:46513 every 8 connection(s)
I20260808 04:04:39.456781 14518 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
I20260808 04:04:39.459312 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 14518
I20260808 04:04:39.459498 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-2/wal/instance
I20260808 04:04:39.460089 5452 external_mini_cluster.cc:1428] Setting key baa27939c46280591b74e961cfbddd1f
Aug 08 04:04:39 dist-test-slave-1l3l krb5kdc[13966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161879, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.83.3@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM
I20260808 04:04:39.475697 14304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:39.459108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.3:54271 (local address 127.5.83.62:39395)
0808 04:04:39.459868 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:39.459872 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:39.460772 (+ 900us) server_negotiation.cc:407] Connection header received
0808 04:04:39.462974 (+ 2202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:39.462980 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:39.463048 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:39.463123 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:39.464579 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:39.465333 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:39.466758 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:39.467145 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:39.467229 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:39.470317 (+ 3088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:39.470322 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:39.470325 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:39.470357 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:39.472887 (+ 2530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:39.473498 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:39.473501 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:39.473503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:39.473641 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:39.474032 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:39.474036 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:39.474038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:39.474302 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:39.474436 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:39.475136 (+ 700us) server_negotiation.cc:299] Negotiation successful
0808 04:04:39.475366 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638}
I20260808 04:04:39.476560 14710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:39.459794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:39395 (local address 127.5.83.3:54271)
0808 04:04:39.460596 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:39.460635 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:39.462698 (+ 2063us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:39.463319 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:39.463327 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:39.463881 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:04:39.464424 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:39.464442 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:39.465457 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:39.465461 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:39.466654 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:39.466666 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:39.467298 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:39.467303 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:39.467487 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:39.468053 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:04:39.468075 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:04:39.470147 (+ 2072us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:04:39.473015 (+ 2868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:39.473021 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:39.473031 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:39.473400 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:39.473735 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:39.473738 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:39.473740 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:39.473902 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:39.474445 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:04:39.474453 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:04:39.474719 (+ 266us) client_negotiation.cc:769] Sending connection context
0808 04:04:39.474944 (+ 225us) client_negotiation.cc:240] Negotiation successful
0808 04:04:39.475415 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":340,"threads_started":1}
I20260808 04:04:39.477713 14708 heartbeater.cc:347] Connected to a master server at 127.5.83.62:39395
I20260808 04:04:39.477996 14708 heartbeater.cc:464] Registering TS with master...
I20260808 04:04:39.478638 14708 heartbeater.cc:511] Master 127.5.83.62:39395 requested a full tablet report, sending...
I20260808 04:04:39.480150 14029 ts_manager.cc:194] Registered new tserver with Master: 4e9480f289294032b1d7d1018594001c (127.5.83.3:46513)
I20260808 04:04:39.480907 14029 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.83.3@KRBTEST.COM'} at 127.5.83.3:54271
I20260808 04:04:39.488001 5452 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.5.83.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.83.61@KRBTEST.COM" created.
I20260808 04:04:39.533592 14301 heartbeater.cc:503] Master 127.5.83.62:39395 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.83.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/krb5kdc/oryx_127.5.83.61.keytab.
Entry for principal oryx/127.5.83.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/krb5kdc/oryx_127.5.83.61.keytab.
I20260808 04:04:39.536609 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.61:41591
--webserver_interface=127.5.83.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/krb5kdc/oryx_127.5.83.61.keytab
--principal=oryx/127.5.83.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:46157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.62:39395,127.5.83.61:41591
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:04:39.855953 14714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:04:39.856334 14714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:04:39.856462 14714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:04:39.863431 14714 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:04:39.863524 14714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:04:39.863598 14714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:04:39.863679 14714 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:04:39.863758 14714 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:04:39.873109 14714 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:46157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.62:39395,127.5.83.61:41591
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/krb5kdc/oryx_127.5.83.61.keytab
--principal=oryx/127.5.83.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.61:41591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.5.83.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
W20260808 04:04:39.873435 14714 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.5.83.62:39395,127.5.83.61:41591'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260808 04:04:39.874423 14714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:04:39.875694 14714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:04:39.886751 14723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:39.886642 14721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:39.886777 14720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:04:39.888078 14714 server_base.cc:1034] running on GCE node
I20260808 04:04:39.888830 14714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:04:39.890178 14714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:04:39.891443 14714 hybrid_clock.cc:648] HybridClock initialized: now 1786161879891378 us; error 42 us; skew 500 ppm
I20260808 04:04:39.891815 14714 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:04:39 dist-test-slave-1l3l krb5kdc[13966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161879, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.83.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:04:39.895009 14714 init.cc:377] Logged in from keytab as oryx/127.5.83.61@KRBTEST.COM (short username oryx)
I20260808 04:04:39.896528 14714 webserver.cc:460] Webserver started at http://127.5.83.61:39525/ using document root <none> and password file <none>
I20260808 04:04:39.897209 14714 fs_manager.cc:359] Metadata directory not provided
I20260808 04:04:39.897311 14714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:04:39.897619 14714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:04:39.900590 14714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/master-1/data/instance:
uuid: "72fecab4e79d4b7baa3fb285de73d640"
format_stamp: "Formatted at 2026-08-08 04:04:39 on dist-test-slave-1l3l"
server_key: "ffb5d660f874a6748c0566523fe6dcce"
server_key_iv: "8c517b5a882abf10fb4c2beb70200e81"
server_key_version: "encryptionkey@0"
I20260808 04:04:39.901325 14714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/master-1/wal/instance:
uuid: "72fecab4e79d4b7baa3fb285de73d640"
format_stamp: "Formatted at 2026-08-08 04:04:39 on dist-test-slave-1l3l"
server_key: "ffb5d660f874a6748c0566523fe6dcce"
server_key_iv: "8c517b5a882abf10fb4c2beb70200e81"
server_key_version: "encryptionkey@0"
I20260808 04:04:39.906244 14714 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 04:04:39.909662 14732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:39.910988 14714 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 04:04:39.911141 14714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/master-1/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/master-1/wal
uuid: "72fecab4e79d4b7baa3fb285de73d640"
format_stamp: "Formatted at 2026-08-08 04:04:39 on dist-test-slave-1l3l"
server_key: "ffb5d660f874a6748c0566523fe6dcce"
server_key_iv: "8c517b5a882abf10fb4c2beb70200e81"
server_key_version: "encryptionkey@0"
I20260808 04:04:39.911317 14714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:04:39.930887 14714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:04:39.934695 14714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:04:39.934924 14714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:04:39.953310 14714 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.61:41591
I20260808 04:04:39.953313 14808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.61:41591 every 8 connection(s)
I20260808 04:04:39.954528 14714 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/master-1/data/info.pb
I20260808 04:04:39.954857 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 14714
I20260808 04:04:39.958673 14809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:04:39.966509 14809 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.83.62" port: 39395 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 04:04:39 dist-test-slave-1l3l krb5kdc[13966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.83.61@KRBTEST.COM
Aug 08 04:04:39 dist-test-slave-1l3l krb5kdc[13966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161879, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.83.61@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM
I20260808 04:04:39.977650 14817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:39.957048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41226 (local address 127.5.83.61:41591)
0808 04:04:39.957764 (+ 716us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:39.957775 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:39.957815 (+ 40us) server_negotiation.cc:407] Connection header received
0808 04:04:39.959466 (+ 1651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:39.959486 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:39.959852 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:39.960154 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:39.961935 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:39.963211 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:39.964944 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:39.965566 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:39.965685 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:39.970504 (+ 4819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:39.970516 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:39.970533 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:39.970577 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:39.973817 (+ 3240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:39.974418 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:39.974424 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:39.974429 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:39.974575 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:39.974933 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:39.974936 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:39.974938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:39.975438 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:39.975737 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:39.976168 (+ 431us) server_negotiation.cc:299] Negotiation successful
0808 04:04:39.976506 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":343,"threads_started":1}
I20260808 04:04:39.983692 14304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:39.969537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33542 (local address 127.5.83.62:39395)
0808 04:04:39.969730 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:39.969736 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:39.970450 (+ 714us) server_negotiation.cc:407] Connection header received
0808 04:04:39.970619 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:39.970625 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:39.970707 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:39.970808 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:39.971977 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:39.973056 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:39.974411 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:39.974822 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:39.974935 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:39.978257 (+ 3322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:39.978262 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:39.978265 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:39.978294 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:39.981109 (+ 2815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:39.981751 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:39.981757 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:39.981761 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:39.981885 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:39.982222 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:39.982227 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:39.982230 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:39.982608 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:39.982743 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:39.983177 (+ 434us) server_negotiation.cc:299] Negotiation successful
0808 04:04:39.983373 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 04:04:39.983760 14819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:39.969524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:39395 (local address 127.0.0.1:33542)
0808 04:04:39.970300 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:39.970339 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:39.970500 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:39.970837 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:39.970841 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:39.971494 (+ 653us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:04:39.971811 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:39.971821 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:39.973189 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:39.973193 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:39.974271 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:39.974283 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:39.975049 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:39.975056 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:39.975182 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:39.975763 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:04:39.975786 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:04:39.978067 (+ 2281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:04:39.981227 (+ 3160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:39.981237 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:39.981244 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:39.981639 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:39.981993 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:39.981997 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:39.982001 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:39.982121 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:39.982729 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:04:39.982736 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:04:39.982889 (+ 153us) client_negotiation.cc:769] Sending connection context
0808 04:04:39.983064 (+ 175us) client_negotiation.cc:240] Negotiation successful
0808 04:04:39.983343 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":422,"threads_started":1}
I20260808 04:04:39.985350 14809 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.83.61" port: 41591 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 04:04:39 dist-test-slave-1l3l krb5kdc[13966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161879, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.83.61@KRBTEST.COM for oryx/127.5.83.61@KRBTEST.COM
I20260808 04:04:39.996256 14819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:39.986107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.61:41591 (local address 127.0.0.1:41228)
0808 04:04:39.986297 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:39.986315 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:39.986406 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:39.986811 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:39.986818 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:39.987103 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:04:39.987224 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:39.987231 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:39.988144 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:39.988147 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:39.988975 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:39.988986 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:39.989586 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:39.989591 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:39.989722 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:39.990143 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:04:39.990157 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:04:39.991903 (+ 1746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:04:39.994405 (+ 2502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:39.994408 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:39.994410 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:39.994753 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:39.995062 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:39.995065 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:39.995067 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:39.995180 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:39.995602 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:04:39.995605 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:04:39.995756 (+ 151us) client_negotiation.cc:769] Sending connection context
0808 04:04:39.995862 (+ 106us) client_negotiation.cc:240] Negotiation successful
0808 04:04:39.996046 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260808 04:04:39.996475 14817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:39.986120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41228 (local address 127.5.83.61:41591)
0808 04:04:39.986570 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:39.986575 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:39.986588 (+ 13us) server_negotiation.cc:407] Connection header received
0808 04:04:39.986623 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:39.986627 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:39.986681 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:39.986763 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:39.987339 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:39.988044 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:39.989087 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:39.989470 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:39.989555 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:39.992062 (+ 2507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:39.992066 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:39.992069 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:39.992096 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:39.994313 (+ 2217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:39.994844 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:39.994847 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:39.994849 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:39.994976 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:39.995259 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:39.995262 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:39.995264 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:39.995514 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:39.995614 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:39.995963 (+ 349us) server_negotiation.cc:299] Negotiation successful
0808 04:04:39.996155 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":365}
I20260808 04:04:40.000751 14809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72fecab4e79d4b7baa3fb285de73d640: Bootstrap starting.
I20260808 04:04:40.003626 14809 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 72fecab4e79d4b7baa3fb285de73d640: Neither blocks nor log segments found. Creating new log.
I20260808 04:04:40.004634 14809 log.cc:824] T 00000000000000000000000000000000 P 72fecab4e79d4b7baa3fb285de73d640: Log is configured to *not* fsync() on all Append() calls
I20260808 04:04:40.007154 14809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72fecab4e79d4b7baa3fb285de73d640: No bootstrap required, opened a new log
I20260808 04:04:40.011138 14504 heartbeater.cc:503] Master 127.5.83.62:39395 was elected leader, sending a full tablet report...
I20260808 04:04:40.014394 14809 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72fecab4e79d4b7baa3fb285de73d640 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b254403e06ed4a0fbad7433dbbb9f425" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 39395 } } peers { permanent_uuid: "72fecab4e79d4b7baa3fb285de73d640" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 41591 } }
I20260808 04:04:40.014724 14809 raft_consensus.cc:374] T 00000000000000000000000000000000 P 72fecab4e79d4b7baa3fb285de73d640 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:04:40.014797 14809 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72fecab4e79d4b7baa3fb285de73d640 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72fecab4e79d4b7baa3fb285de73d640, State: Initialized, Role: FOLLOWER
I20260808 04:04:40.015354 14809 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72fecab4e79d4b7baa3fb285de73d640 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b254403e06ed4a0fbad7433dbbb9f425" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 39395 } } peers { permanent_uuid: "72fecab4e79d4b7baa3fb285de73d640" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 41591 } }
I20260808 04:04:40.016147 14822 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72fecab4e79d4b7baa3fb285de73d640 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b254403e06ed4a0fbad7433dbbb9f425" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 39395 } } peers { permanent_uuid: "72fecab4e79d4b7baa3fb285de73d640" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 41591 } } }
I20260808 04:04:40.016427 14822 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72fecab4e79d4b7baa3fb285de73d640 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:04:40.016978 14809 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72fecab4e79d4b7baa3fb285de73d640 [sys.catalog]: configured and running, proceeding with master startup.
W20260808 04:04:40.025817 14834 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 72fecab4e79d4b7baa3fb285de73d640: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 04:04:40.025945 14834 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 04:04:40.029160 14714 master_runner.cc:428] Detected that this master 72fecab4e79d4b7baa3fb285de73d640 is joining an existing cluster
I20260808 04:04:40.029255 14714 master_runner.cc:432] Initiating AddMaster RPC to add 127.5.83.61:41591
I20260808 04:04:40.058979 14817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:40.033617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41234 (local address 127.5.83.61:41591)
0808 04:04:40.033856 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:40.033862 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:40.044308 (+ 10446us) server_negotiation.cc:407] Connection header received
0808 04:04:40.044482 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:40.044488 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:40.044566 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:40.044670 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:40.046176 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:40.047230 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:40.050159 (+ 2929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:40.050595 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:40.050705 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:40.052709 (+ 2004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:40.052715 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:40.052719 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:40.052751 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:40.055923 (+ 3172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:40.056669 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:40.056674 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:40.056677 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:40.056816 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:40.057219 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:40.057223 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:40.057226 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:40.057647 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:40.057784 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:40.058416 (+ 632us) server_negotiation.cc:299] Negotiation successful
0808 04:04:40.058630 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260808 04:04:40.059944 14845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:40.033926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.61:41591 (local address 127.0.0.1:41234)
0808 04:04:40.044186 (+ 10260us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:40.044208 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:40.044363 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:40.044842 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:40.044847 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:40.045621 (+ 774us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:04:40.046044 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:40.046061 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:40.048499 (+ 2438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:40.048504 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:40.050043 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:40.050053 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:40.050765 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:40.050771 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:40.050939 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:40.051672 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:04:40.051694 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:04:40.052552 (+ 858us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:04:40.056056 (+ 3504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:40.056061 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:40.056064 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:40.056556 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:40.056940 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:40.056944 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:40.056947 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:40.057118 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:40.057846 (+ 728us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:04:40.057850 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:04:40.058108 (+ 258us) client_negotiation.cc:769] Sending connection context
0808 04:04:40.059252 (+ 1144us) client_negotiation.cc:240] Negotiation successful
0808 04:04:40.059522 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10099,"thread_start_us":15434,"threads_started":1}
I20260808 04:04:40.066612 14841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:40.034416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:39395 (local address 127.0.0.1:33554)
0808 04:04:40.049689 (+ 15273us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:40.049708 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:40.049820 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:40.050920 (+ 1100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:40.050925 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:40.051535 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:04:40.051880 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:40.051894 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:40.053420 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:40.053426 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:40.055836 (+ 2410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:40.055851 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:40.057033 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:40.057041 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:40.057204 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:40.057814 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:04:40.057837 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:04:40.058876 (+ 1039us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:04:40.063827 (+ 4951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:40.063831 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:40.063834 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:40.064333 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:40.064766 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:40.064771 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:40.064774 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:40.064923 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:40.065554 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:04:40.065558 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:04:40.065755 (+ 197us) client_negotiation.cc:769] Sending connection context
0808 04:04:40.065905 (+ 150us) client_negotiation.cc:240] Negotiation successful
0808 04:04:40.066202 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15131,"thread_start_us":9440,"threads_started":1}
I20260808 04:04:40.066645 14304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:40.033614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33554 (local address 127.5.83.62:39395)
0808 04:04:40.034012 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:40.034018 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:40.049781 (+ 15763us) server_negotiation.cc:407] Connection header received
0808 04:04:40.049941 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:40.049947 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:40.050029 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:40.050146 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:40.052048 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:40.053214 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:40.056007 (+ 2793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:40.056643 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:40.056741 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:40.059100 (+ 2359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:40.059106 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:40.059111 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:40.059150 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:40.063157 (+ 4007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:40.064477 (+ 1320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:40.064483 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:40.064487 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:40.064649 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:40.065046 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:40.065051 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:40.065054 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:40.065423 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:40.065569 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:40.066050 (+ 481us) server_negotiation.cc:299] Negotiation successful
0808 04:04:40.066279 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284}
WARNING: no policy specified for oryx/127.5.83.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.83.60@KRBTEST.COM" created.
Aug 08 04:04:40 dist-test-slave-1l3l krb5kdc[13966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161877, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.83.62@KRBTEST.COM for oryx/127.5.83.61@KRBTEST.COM
I20260808 04:04:40.094566 14847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:40.073302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.61:41591 (local address 127.0.0.1:41240)
0808 04:04:40.073946 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:40.073999 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:40.074236 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:40.074693 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:40.074700 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:40.075568 (+ 868us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:04:40.075869 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:40.075882 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:40.077181 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:40.077188 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:40.078779 (+ 1591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:40.078795 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:40.079926 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:40.079931 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:40.080108 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:40.080726 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:04:40.080747 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:04:40.083650 (+ 2903us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:04:40.091647 (+ 7997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:40.091659 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:40.091667 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:40.092263 (+ 596us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:40.092694 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:40.092699 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:40.092703 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:40.092866 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:40.093503 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:04:40.093511 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:04:40.093715 (+ 204us) client_negotiation.cc:769] Sending connection context
0808 04:04:40.093905 (+ 190us) client_negotiation.cc:240] Negotiation successful
0808 04:04:40.094248 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":276,"threads_started":1}
I20260808 04:04:40.094623 14817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:40.073280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41240 (local address 127.5.83.61:41591)
0808 04:04:40.073463 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:40.073468 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:40.074093 (+ 625us) server_negotiation.cc:407] Connection header received
0808 04:04:40.074442 (+ 349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:40.074448 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:40.074532 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:40.074632 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:40.076017 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:40.077037 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:40.078954 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:40.079569 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:40.079736 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:40.087896 (+ 8160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:40.087903 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:40.087908 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:40.087946 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:40.091492 (+ 3546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:40.092406 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:40.092412 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:40.092416 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:40.092567 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:40.092989 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:40.092994 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:40.093001 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:40.093357 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:40.093495 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:40.094042 (+ 547us) server_negotiation.cc:299] Negotiation successful
0808 04:04:40.094257 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260808 04:04:40.096593 14029 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.5.83.61:41591: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "72fecab4e79d4b7baa3fb285de73d640"
member_type: NON_VOTER
last_known_addr {
host: "127.5.83.61"
port: 41591
}
attrs {
promote: true
}
}
dest_uuid: "b254403e06ed4a0fbad7433dbbb9f425"
cas_config_opid_index: -1
I20260808 04:04:40.098321 14029 consensus_queue.cc:237] T 00000000000000000000000000000000 P b254403e06ed4a0fbad7433dbbb9f425 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b254403e06ed4a0fbad7433dbbb9f425" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 39395 } } peers { permanent_uuid: "72fecab4e79d4b7baa3fb285de73d640" member_type: NON_VOTER last_known_addr { host: "127.5.83.61" port: 41591 } attrs { promote: true } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.83.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/krb5kdc/oryx_127.5.83.60.keytab.
Entry for principal oryx/127.5.83.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/krb5kdc/oryx_127.5.83.60.keytab.
I20260808 04:04:40.101735 14854 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b254403e06ed4a0fbad7433dbbb9f425 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 72fecab4e79d4b7baa3fb285de73d640 (127.5.83.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b254403e06ed4a0fbad7433dbbb9f425" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 39395 } } peers { permanent_uuid: "72fecab4e79d4b7baa3fb285de73d640" member_type: NON_VOTER last_known_addr { host: "127.5.83.61" port: 41591 } attrs { promote: true } } }
I20260808 04:04:40.103129 14767 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72fecab4e79d4b7baa3fb285de73d640 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:04:40.103129 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.60:34729
--webserver_interface=127.5.83.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/krb5kdc/oryx_127.5.83.60.keytab
--principal=oryx/127.5.83.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:46157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.62:39395,127.5.83.61:41591,127.5.83.60:34729
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:04:40.106209 14854 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.5.83.61:41591
I20260808 04:04:40.106556 14767 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 72fecab4e79d4b7baa3fb285de73d640 [term 1 FOLLOWER]: Refusing update from remote peer b254403e06ed4a0fbad7433dbbb9f425: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260808 04:04:40.107149 14714 master.cc:537] Master@127.5.83.61:41591 shutting down...
I20260808 04:04:40.108561 14854 sys_catalog.cc:455] T 00000000000000000000000000000000 P b254403e06ed4a0fbad7433dbbb9f425 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b254403e06ed4a0fbad7433dbbb9f425" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b254403e06ed4a0fbad7433dbbb9f425" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 39395 } } peers { permanent_uuid: "72fecab4e79d4b7baa3fb285de73d640" member_type: NON_VOTER last_known_addr { host: "127.5.83.61" port: 41591 } attrs { promote: true } } }
I20260808 04:04:40.108789 14854 sys_catalog.cc:458] T 00000000000000000000000000000000 P b254403e06ed4a0fbad7433dbbb9f425 [sys.catalog]: This master's current role is: LEADER
I20260808 04:04:40.116178 14853 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b254403e06ed4a0fbad7433dbbb9f425 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72fecab4e79d4b7baa3fb285de73d640" member_type: NON_VOTER last_known_addr { host: "127.5.83.61" port: 41591 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 04:04:40.119870 14714 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 72fecab4e79d4b7baa3fb285de73d640 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:04:40.120265 14714 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 72fecab4e79d4b7baa3fb285de73d640 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:04:40.120497 14714 tablet_replica.cc:330] T 00000000000000000000000000000000 P 72fecab4e79d4b7baa3fb285de73d640: stopping tablet replica
W20260808 04:04:40.123270 14012 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260808 04:04:40.128485 14714 master.cc:552] Master@127.5.83.61:41591 shutdown complete.
I20260808 04:04:40.128607 14714 master_runner.cc:305] Clearing existing system tablet
I20260808 04:04:40.129140 14714 fs_manager.cc:359] Metadata directory not provided
I20260808 04:04:40.129254 14714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:04:40.131450 14847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:40.130982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.61:41591 (local address 127.0.0.1:41254)
0808 04:04:40.131144 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:40.131258 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.61:41591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260808 04:04:40.131875 14012 consensus_peers.cc:487] T 00000000000000000000000000000000 P b254403e06ed4a0fbad7433dbbb9f425 -> Peer 72fecab4e79d4b7baa3fb285de73d640 (127.5.83.61:41591): Couldn't send request to peer 72fecab4e79d4b7baa3fb285de73d640. Status: Network error: Client connection negotiation failed: client connection to 127.5.83.61:41591: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260808 04:04:40.135139 14714 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 04:04:40.136937 14865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:40.137240 14714 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:04:40.137518 14714 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 04:04:40.137606 14714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/master-1/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/master-1/wal
uuid: "72fecab4e79d4b7baa3fb285de73d640"
format_stamp: "Formatted at 2026-08-08 04:04:39 on dist-test-slave-1l3l"
server_key: "ffb5d660f874a6748c0566523fe6dcce"
server_key_iv: "8c517b5a882abf10fb4c2beb70200e81"
server_key_version: "encryptionkey@0"
I20260808 04:04:40.142969 14714 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 72fecab4e79d4b7baa3fb285de73d640: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:04:40.149968 14714 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 72fecab4e79d4b7baa3fb285de73d640: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260808 04:04:40.150142 14714 log.cc:1196] T 00000000000000000000000000000000 P 72fecab4e79d4b7baa3fb285de73d640: Deleting WAL directory at /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260808 04:04:40.150588 14714 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 72fecab4e79d4b7baa3fb285de73d640: Deleting consensus metadata
I20260808 04:04:40.150843 14714 master_runner.cc:315] Copying system tablet from 127.5.83.62:39395
I20260808 04:04:40.157595 14714 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 72fecab4e79d4b7baa3fb285de73d640: tablet copy: Beginning tablet copy session from remote peer at address 127.5.83.62:39395
I20260808 04:04:40.174410 14304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:40.158620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33560 (local address 127.5.83.62:39395)
0808 04:04:40.158814 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:40.158821 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:40.160291 (+ 1470us) server_negotiation.cc:407] Connection header received
0808 04:04:40.160452 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:40.160459 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:40.160561 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:40.160672 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:40.161718 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:40.162817 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:40.164634 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:40.165304 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:40.165501 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:40.167778 (+ 2277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:40.167786 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:40.167791 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:40.167844 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:40.171501 (+ 3657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:40.172308 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:40.172313 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:40.172317 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:40.172477 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:40.172889 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:40.172894 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:40.172897 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:40.173242 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:40.173395 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:40.173846 (+ 451us) server_negotiation.cc:299] Negotiation successful
0808 04:04:40.174080 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 04:04:40.174503 14876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:40.158613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:39395 (local address 127.0.0.1:33560)
0808 04:04:40.160074 (+ 1461us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:40.160100 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:40.160256 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:40.160715 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:40.160720 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:40.161261 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:04:40.161555 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:40.161568 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:40.162967 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:40.162980 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:40.164475 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:40.164490 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:40.165526 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:40.165532 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:40.165703 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:40.166454 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:04:40.166478 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:04:40.167497 (+ 1019us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:04:40.171637 (+ 4140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:40.171643 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:40.171647 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:40.172169 (+ 522us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:40.172592 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:40.172598 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:40.172602 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:40.172770 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:40.173375 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:04:40.173380 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:04:40.173549 (+ 169us) client_negotiation.cc:769] Sending connection context
0808 04:04:40.173692 (+ 143us) client_negotiation.cc:240] Negotiation successful
0808 04:04:40.174038 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1327,"thread_start_us":1236,"threads_started":1}
I20260808 04:04:40.176012 14066 tablet_copy_service.cc:140] P b254403e06ed4a0fbad7433dbbb9f425: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 72fecab4e79d4b7baa3fb285de73d640 ({username='oryx', principal='oryx/127.5.83.61@KRBTEST.COM'} at 127.0.0.1:33560)
I20260808 04:04:40.176440 14066 tablet_copy_service.cc:161] P b254403e06ed4a0fbad7433dbbb9f425: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 72fecab4e79d4b7baa3fb285de73d640 at {username='oryx', principal='oryx/127.5.83.61@KRBTEST.COM'} at 127.0.0.1:33560: session id = 72fecab4e79d4b7baa3fb285de73d640-00000000000000000000000000000000
I20260808 04:04:40.181391 14066 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P b254403e06ed4a0fbad7433dbbb9f425: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 04:04:40.184082 14714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260808 04:04:40.190964 14714 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 72fecab4e79d4b7baa3fb285de73d640: tablet copy: Starting download of 0 data blocks...
I20260808 04:04:40.191336 14714 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 72fecab4e79d4b7baa3fb285de73d640: tablet copy: Starting download of 1 WAL segments...
I20260808 04:04:40.194955 14714 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 72fecab4e79d4b7baa3fb285de73d640: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 04:04:40.199321 14066 tablet_copy_service.cc:342] P b254403e06ed4a0fbad7433dbbb9f425: Request end of tablet copy session 72fecab4e79d4b7baa3fb285de73d640-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.5.83.61@KRBTEST.COM'} at 127.0.0.1:33560
I20260808 04:04:40.199579 14066 tablet_copy_service.cc:433] P b254403e06ed4a0fbad7433dbbb9f425: ending tablet copy session 72fecab4e79d4b7baa3fb285de73d640-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 72fecab4e79d4b7baa3fb285de73d640
I20260808 04:04:40.211781 14714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:04:40.214435 14877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:40.221519 14879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:40.222000 14881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:04:40.222383 14714 server_base.cc:1034] running on GCE node
I20260808 04:04:40.222690 14714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:04:40.223242 14714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:04:40.224452 14714 hybrid_clock.cc:648] HybridClock initialized: now 1786161880224385 us; error 36 us; skew 500 ppm
I20260808 04:04:40.224673 14714 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:04:40 dist-test-slave-1l3l krb5kdc[13966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161880, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.83.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:04:40.226495 14714 init.cc:377] Logged in from keytab as oryx/127.5.83.61@KRBTEST.COM (short username oryx)
I20260808 04:04:40.227659 14714 webserver.cc:460] Webserver started at http://127.5.83.61:40223/ using document root <none> and password file <none>
I20260808 04:04:40.227967 14714 fs_manager.cc:359] Metadata directory not provided
I20260808 04:04:40.228044 14714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:04:40.230492 14714 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20260808 04:04:40.232125 14890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:40.232488 14714 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 04:04:40.232623 14714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/master-1/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/master-1/wal
uuid: "72fecab4e79d4b7baa3fb285de73d640"
format_stamp: "Formatted at 2026-08-08 04:04:39 on dist-test-slave-1l3l"
server_key: "ffb5d660f874a6748c0566523fe6dcce"
server_key_iv: "8c517b5a882abf10fb4c2beb70200e81"
server_key_version: "encryptionkey@0"
I20260808 04:04:40.232761 14714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:04:40.256513 14714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:04:40.259621 14714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:04:40.289930 14714 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.61:41591
I20260808 04:04:40.289950 14963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.61:41591 every 8 connection(s)
I20260808 04:04:40.290824 14714 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/master-1/data/info.pb
I20260808 04:04:40.296468 14965 sys_catalog.cc:263] Verifying existing consensus state
I20260808 04:04:40.300758 14965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72fecab4e79d4b7baa3fb285de73d640: Bootstrap starting.
I20260808 04:04:40.315093 14965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72fecab4e79d4b7baa3fb285de73d640: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 04:04:40.315553 14965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72fecab4e79d4b7baa3fb285de73d640: Bootstrap complete.
I20260808 04:04:40.316387 14965 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72fecab4e79d4b7baa3fb285de73d640 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b254403e06ed4a0fbad7433dbbb9f425" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 39395 } } peers { permanent_uuid: "72fecab4e79d4b7baa3fb285de73d640" member_type: NON_VOTER last_known_addr { host: "127.5.83.61" port: 41591 } attrs { promote: true } }
I20260808 04:04:40.316596 14965 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72fecab4e79d4b7baa3fb285de73d640 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 72fecab4e79d4b7baa3fb285de73d640, State: Initialized, Role: LEARNER
I20260808 04:04:40.316818 14965 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72fecab4e79d4b7baa3fb285de73d640 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b254403e06ed4a0fbad7433dbbb9f425" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 39395 } } peers { permanent_uuid: "72fecab4e79d4b7baa3fb285de73d640" member_type: NON_VOTER last_known_addr { host: "127.5.83.61" port: 41591 } attrs { promote: true } }
I20260808 04:04:40.317574 14969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72fecab4e79d4b7baa3fb285de73d640 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b254403e06ed4a0fbad7433dbbb9f425" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 39395 } } peers { permanent_uuid: "72fecab4e79d4b7baa3fb285de73d640" member_type: NON_VOTER last_known_addr { host: "127.5.83.61" port: 41591 } attrs { promote: true } } }
I20260808 04:04:40.317860 14965 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72fecab4e79d4b7baa3fb285de73d640 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:04:40.318010 14969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72fecab4e79d4b7baa3fb285de73d640 [sys.catalog]: This master's current role is: LEARNER
I20260808 04:04:40.331581 14984 catalog_manager.cc:1259] Loaded cluster ID: a15e7f4b3b484d4dbf31a2eea6d58df7
I20260808 04:04:40.331665 14984 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 72fecab4e79d4b7baa3fb285de73d640: loading cluster ID for follower catalog manager: success
I20260808 04:04:40.334443 14984 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 72fecab4e79d4b7baa3fb285de73d640: acquiring CA information for follower catalog manager: success
I20260808 04:04:40.336570 14984 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 72fecab4e79d4b7baa3fb285de73d640: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 04:04:40.483064 14708 heartbeater.cc:503] Master 127.5.83.62:39395 was elected leader, sending a full tablet report...
W20260808 04:04:40.560218 14860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:04:40.560528 14860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:04:40.560662 14860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:04:40.568197 14860 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:04:40.568310 14860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:04:40.568385 14860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:04:40.568502 14860 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:04:40.568575 14860 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:04:40.580042 14860 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:46157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.62:39395,127.5.83.61:41591,127.5.83.60:34729
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/krb5kdc/oryx_127.5.83.60.keytab
--principal=oryx/127.5.83.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.60:34729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.5.83.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:04:40.581365 14860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:04:40.582643 14860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:04:40.594275 14990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:40.595016 14995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:40.595073 14991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:04:40.596007 14860 server_base.cc:1034] running on GCE node
I20260808 04:04:40.596706 14860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:04:40.598400 14860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:04:40.599619 14860 hybrid_clock.cc:648] HybridClock initialized: now 1786161880599583 us; error 24 us; skew 500 ppm
I20260808 04:04:40.599984 14860 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:04:40 dist-test-slave-1l3l krb5kdc[13966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161880, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.83.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:04:40.606870 14860 init.cc:377] Logged in from keytab as oryx/127.5.83.60@KRBTEST.COM (short username oryx)
I20260808 04:04:40.608430 14860 webserver.cc:460] Webserver started at http://127.5.83.60:39167/ using document root <none> and password file <none>
I20260808 04:04:40.609251 14860 fs_manager.cc:359] Metadata directory not provided
I20260808 04:04:40.609486 14860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:04:40.609876 14860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:04:40.613112 14860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/master-2/data/instance:
uuid: "f0960abfe8684bd79c949f4811b75825"
format_stamp: "Formatted at 2026-08-08 04:04:40 on dist-test-slave-1l3l"
server_key: "a68b794425ead5f714388c4a5ac41712"
server_key_iv: "4523951aaf975e745cc19ada88f527af"
server_key_version: "encryptionkey@0"
I20260808 04:04:40.613865 14860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/master-2/wal/instance:
uuid: "f0960abfe8684bd79c949f4811b75825"
format_stamp: "Formatted at 2026-08-08 04:04:40 on dist-test-slave-1l3l"
server_key: "a68b794425ead5f714388c4a5ac41712"
server_key_iv: "4523951aaf975e745cc19ada88f527af"
server_key_version: "encryptionkey@0"
I20260808 04:04:40.619030 14860 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 04:04:40.622692 15003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:40.624027 14860 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:04:40.624174 14860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/master-2/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/master-2/wal
uuid: "f0960abfe8684bd79c949f4811b75825"
format_stamp: "Formatted at 2026-08-08 04:04:40 on dist-test-slave-1l3l"
server_key: "a68b794425ead5f714388c4a5ac41712"
server_key_iv: "4523951aaf975e745cc19ada88f527af"
server_key_version: "encryptionkey@0"
I20260808 04:04:40.624344 14860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:04:40.638521 14860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:04:40.643813 14860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:04:40.644083 14860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:04:40.646761 15010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:40.638730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.61:41591 (local address 127.0.0.1:41264)
0808 04:04:40.639164 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:40.639193 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:40.639349 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:40.640219 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:40.640223 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:40.640280 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:04:40.640465 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:40.640473 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:40.642665 (+ 2192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:40.642669 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:40.644116 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:40.644129 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:40.645886 (+ 1757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:40.645892 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:40.646081 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:40.646152 (+ 71us) client_negotiation.cc:769] Sending connection context
0808 04:04:40.646264 (+ 112us) client_negotiation.cc:240] Negotiation successful
0808 04:04:40.646430 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":219,"threads_started":1}
I20260808 04:04:40.647277 15011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:40.638914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41264 (local address 127.5.83.61:41591)
0808 04:04:40.639766 (+ 852us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:40.639773 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:40.639796 (+ 23us) server_negotiation.cc:407] Connection header received
0808 04:04:40.639874 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:40.639878 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:40.640018 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:40.640152 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:04:40.640687 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:40.642521 (+ 1834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:40.644291 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:40.645647 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:40.646661 (+ 1014us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:40.646847 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:40.646908 (+ 61us) server_negotiation.cc:299] Negotiation successful
0808 04:04:40.647039 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":697,"thread_start_us":503,"threads_started":1}
I20260808 04:04:40.648502 14925 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 72fecab4e79d4b7baa3fb285de73d640 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260808 04:04:40.649192 14969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72fecab4e79d4b7baa3fb285de73d640 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b254403e06ed4a0fbad7433dbbb9f425. Latest consensus state: current_term: 1 leader_uuid: "b254403e06ed4a0fbad7433dbbb9f425" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b254403e06ed4a0fbad7433dbbb9f425" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 39395 } } peers { permanent_uuid: "72fecab4e79d4b7baa3fb285de73d640" member_type: NON_VOTER last_known_addr { host: "127.5.83.61" port: 41591 } attrs { promote: true } } }
I20260808 04:04:40.649461 14969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72fecab4e79d4b7baa3fb285de73d640 [sys.catalog]: This master's current role is: LEARNER
I20260808 04:04:40.667980 14860 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.60:34729
I20260808 04:04:40.668010 15076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.60:34729 every 8 connection(s)
I20260808 04:04:40.669322 14860 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/master-2/data/info.pb
I20260808 04:04:40.673069 15077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:04:40.676184 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 14860
I20260808 04:04:40.680706 15077 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.83.62" port: 39395 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 04:04:40 dist-test-slave-1l3l krb5kdc[13966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.83.60@KRBTEST.COM
Aug 08 04:04:40 dist-test-slave-1l3l krb5kdc[13966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161880, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.83.60@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM
I20260808 04:04:40.696808 15081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:40.683054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33564 (local address 127.5.83.62:39395)
0808 04:04:40.683523 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:40.683527 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:40.684169 (+ 642us) server_negotiation.cc:407] Connection header received
0808 04:04:40.684322 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:40.684327 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:40.684386 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:40.684474 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:40.685535 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:40.686298 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:40.687862 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:40.688306 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:40.688419 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:40.691465 (+ 3046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:40.691470 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:40.691473 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:40.691505 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:40.694101 (+ 2596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:40.694764 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:40.694768 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:40.694770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:40.694886 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:40.695255 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:40.695260 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:40.695263 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:40.695535 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:40.695665 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:40.696210 (+ 545us) server_negotiation.cc:299] Negotiation successful
0808 04:04:40.696429 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":245,"threads_started":1}
I20260808 04:04:40.697772 15080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:40.678228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59244 (local address 127.5.83.60:34729)
0808 04:04:40.678991 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:40.679003 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:40.679042 (+ 39us) server_negotiation.cc:407] Connection header received
0808 04:04:40.680705 (+ 1663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:40.680727 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:40.681098 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:40.681436 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:40.683150 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:40.684885 (+ 1735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:40.686153 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:40.686808 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:40.686985 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:40.691327 (+ 4342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:40.691336 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:40.691347 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:40.691382 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:40.694107 (+ 2725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:40.694729 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:40.694735 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:40.694737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:40.694891 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:40.695248 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:40.695252 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:40.695254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:40.695773 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:40.696062 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:40.696363 (+ 301us) server_negotiation.cc:299] Negotiation successful
0808 04:04:40.696763 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":130,"server-negotiator.queue_time_us":559,"thread_start_us":365,"threads_started":1}
I20260808 04:04:40.697849 15083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:40.682975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:39395 (local address 127.0.0.1:33564)
0808 04:04:40.683985 (+ 1010us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:40.684062 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:40.684210 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:40.684548 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:40.684555 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:40.685151 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:04:40.685388 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:40.685406 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:40.686437 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:40.686441 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:40.687728 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:40.687743 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:40.688522 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:40.688530 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:40.688664 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:40.689217 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:04:40.689238 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:04:40.691283 (+ 2045us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:04:40.694219 (+ 2936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:40.694230 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:40.694241 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:40.694655 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:40.695001 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:40.695006 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:40.695008 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:40.695143 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:40.695665 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:04:40.695672 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:04:40.695881 (+ 209us) client_negotiation.cc:769] Sending connection context
0808 04:04:40.696116 (+ 235us) client_negotiation.cc:240] Negotiation successful
0808 04:04:40.696599 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":820,"thread_start_us":653,"threads_started":1}
I20260808 04:04:40.699290 15077 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.83.61" port: 41591 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 04:04:40 dist-test-slave-1l3l krb5kdc[13966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161880, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.83.60@KRBTEST.COM for oryx/127.5.83.61@KRBTEST.COM
I20260808 04:04:40.711665 15083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:40.700064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.61:41591 (local address 127.0.0.1:41268)
0808 04:04:40.700270 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:40.700288 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:40.700455 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:40.700765 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:40.700769 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:40.701120 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:04:40.701247 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:40.701255 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:40.702235 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:40.702241 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:40.703153 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:40.703164 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:40.703865 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:40.703870 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:40.704059 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:40.704586 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:04:40.704603 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:04:40.706571 (+ 1968us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:04:40.709401 (+ 2830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:40.709406 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:40.709410 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:40.709785 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:40.710173 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:40.710177 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:40.710180 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:40.710314 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:40.710813 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:04:40.710818 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:04:40.710951 (+ 133us) client_negotiation.cc:769] Sending connection context
0808 04:04:40.711064 (+ 113us) client_negotiation.cc:240] Negotiation successful
0808 04:04:40.711336 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260808 04:04:40.711745 15011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:40.700124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41268 (local address 127.5.83.61:41591)
0808 04:04:40.700301 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:40.700306 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:40.700409 (+ 103us) server_negotiation.cc:407] Connection header received
0808 04:04:40.700580 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:40.700586 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:40.700650 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:40.700733 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:40.701395 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:40.702109 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:40.703288 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:40.703730 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:40.703857 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:40.706751 (+ 2894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:40.706756 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:40.706759 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:40.706790 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:40.709265 (+ 2475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:40.709900 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:40.709904 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:40.709906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:40.710065 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:40.710417 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:40.710421 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:40.710424 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:40.710702 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:40.710829 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:40.711168 (+ 339us) server_negotiation.cc:299] Negotiation successful
0808 04:04:40.711407 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260808 04:04:40.712647 15077 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.83.60" port: 34729 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 04:04:40 dist-test-slave-1l3l krb5kdc[13966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161880, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.83.60@KRBTEST.COM for oryx/127.5.83.60@KRBTEST.COM
I20260808 04:04:40.724185 15083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:40.713241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.60:34729 (local address 127.0.0.1:59246)
0808 04:04:40.713465 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:40.713479 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:40.713560 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:40.713851 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:40.713855 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:40.714162 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:04:40.714297 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:40.714304 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:40.715183 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:40.715186 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:40.716098 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:40.716110 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:40.716748 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:40.716752 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:40.716927 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:40.717361 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:04:40.717384 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:04:40.719239 (+ 1855us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:04:40.722080 (+ 2841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:40.722084 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:40.722087 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:40.722445 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:40.722818 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:40.722821 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:40.722823 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:40.722971 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:40.723466 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:04:40.723470 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:04:40.723585 (+ 115us) client_negotiation.cc:769] Sending connection context
0808 04:04:40.723692 (+ 107us) client_negotiation.cc:240] Negotiation successful
0808 04:04:40.723922 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260808 04:04:40.724406 15080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:40.713270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59246 (local address 127.5.83.60:34729)
0808 04:04:40.713461 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:40.713465 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:40.713529 (+ 64us) server_negotiation.cc:407] Connection header received
0808 04:04:40.713651 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:40.713654 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:40.713705 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:40.713775 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:40.714430 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:40.715066 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:40.716213 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:40.716615 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:40.716712 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:40.719397 (+ 2685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:40.719403 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:40.719408 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:40.719436 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:40.721960 (+ 2524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:40.722535 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:40.722539 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:40.722541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:40.722712 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:40.723061 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:40.723065 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:40.723067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:40.723365 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:40.723473 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:40.723819 (+ 346us) server_negotiation.cc:299] Negotiation successful
0808 04:04:40.724052 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":51}
I20260808 04:04:40.729161 15077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0960abfe8684bd79c949f4811b75825: Bootstrap starting.
I20260808 04:04:40.732213 15077 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0960abfe8684bd79c949f4811b75825: Neither blocks nor log segments found. Creating new log.
I20260808 04:04:40.733268 15077 log.cc:824] T 00000000000000000000000000000000 P f0960abfe8684bd79c949f4811b75825: Log is configured to *not* fsync() on all Append() calls
I20260808 04:04:40.736004 15077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0960abfe8684bd79c949f4811b75825: No bootstrap required, opened a new log
I20260808 04:04:40.743284 15077 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0960abfe8684bd79c949f4811b75825 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b254403e06ed4a0fbad7433dbbb9f425" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 39395 } } peers { permanent_uuid: "72fecab4e79d4b7baa3fb285de73d640" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 41591 } } peers { permanent_uuid: "f0960abfe8684bd79c949f4811b75825" member_type: VOTER last_known_addr { host: "127.5.83.60" port: 34729 } }
I20260808 04:04:40.743568 15077 raft_consensus.cc:374] T 00000000000000000000000000000000 P f0960abfe8684bd79c949f4811b75825 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:04:40.743675 15077 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0960abfe8684bd79c949f4811b75825 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0960abfe8684bd79c949f4811b75825, State: Initialized, Role: FOLLOWER
I20260808 04:04:40.744247 15077 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0960abfe8684bd79c949f4811b75825 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b254403e06ed4a0fbad7433dbbb9f425" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 39395 } } peers { permanent_uuid: "72fecab4e79d4b7baa3fb285de73d640" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 41591 } } peers { permanent_uuid: "f0960abfe8684bd79c949f4811b75825" member_type: VOTER last_known_addr { host: "127.5.83.60" port: 34729 } }
I20260808 04:04:40.745071 15085 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0960abfe8684bd79c949f4811b75825 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b254403e06ed4a0fbad7433dbbb9f425" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 39395 } } peers { permanent_uuid: "72fecab4e79d4b7baa3fb285de73d640" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 41591 } } peers { permanent_uuid: "f0960abfe8684bd79c949f4811b75825" member_type: VOTER last_known_addr { host: "127.5.83.60" port: 34729 } } }
I20260808 04:04:40.745430 15085 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0960abfe8684bd79c949f4811b75825 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:04:40.746037 15077 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0960abfe8684bd79c949f4811b75825 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:04:40.752522 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 14121
W20260808 04:04:40.755247 15098 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f0960abfe8684bd79c949f4811b75825: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 04:04:40.755478 15098 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 04:04:40.760391 14860 master_runner.cc:428] Detected that this master f0960abfe8684bd79c949f4811b75825 is joining an existing cluster
I20260808 04:04:40.760494 14860 master_runner.cc:432] Initiating AddMaster RPC to add 127.5.83.60:34729
Aug 08 04:04:40 dist-test-slave-1l3l krb5kdc[13966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.83.1@KRBTEST.COM
I20260808 04:04:40.773655 15101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:40.753823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36942 (local address 127.5.83.1:45257)
0808 04:04:40.754372 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:40.754391 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:40.754419 (+ 28us) server_negotiation.cc:407] Connection header received
0808 04:04:40.754497 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:40.754506 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:40.754736 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:40.755763 (+ 1027us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:40.756536 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:40.758117 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:40.759676 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:40.760528 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:40.760660 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:40.766383 (+ 5723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:40.766399 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:40.766414 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:40.766469 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:40.770187 (+ 3718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:40.771021 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:40.771030 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:40.771034 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:40.771179 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:40.771619 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:40.771625 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:40.771629 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:40.772081 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:40.772343 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:40.772977 (+ 634us) server_negotiation.cc:299] Negotiation successful
0808 04:04:40.773238 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":1974,"threads_started":1}
I20260808 04:04:40.776098 14294 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:04:40.781567 15011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:40.764986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41282 (local address 127.5.83.61:41591)
0808 04:04:40.765174 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:40.765180 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:40.766707 (+ 1527us) server_negotiation.cc:407] Connection header received
0808 04:04:40.766947 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:40.766953 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:40.767029 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:40.767099 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:40.768645 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:40.769729 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:40.771850 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:40.772458 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:40.772542 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:40.774913 (+ 2371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:40.774919 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:40.774923 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:40.774960 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:40.777912 (+ 2952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:40.778627 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:40.778632 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:40.778635 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:40.778783 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:40.779214 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:40.779219 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:40.779222 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:40.779553 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:40.779737 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:40.781001 (+ 1264us) server_negotiation.cc:299] Negotiation successful
0808 04:04:40.781237 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 04:04:40.781715 15108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:40.764941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.61:41591 (local address 127.0.0.1:41282)
0808 04:04:40.766590 (+ 1649us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:40.766618 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:40.766810 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:40.767308 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:40.767312 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:40.768071 (+ 759us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:04:40.768474 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:40.768489 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:40.769964 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:40.769969 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:40.771672 (+ 1703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:40.771687 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:40.772828 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:40.772835 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:40.773011 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:40.773647 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:04:40.773669 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:04:40.774712 (+ 1043us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:04:40.778029 (+ 3317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:40.778034 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:40.778037 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:40.778488 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:40.778915 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:40.778920 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:40.778923 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:40.779096 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:40.779798 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:04:40.779803 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:04:40.780036 (+ 233us) client_negotiation.cc:769] Sending connection context
0808 04:04:40.780233 (+ 197us) client_negotiation.cc:240] Negotiation successful
0808 04:04:40.781255 (+ 1022us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1501,"mutex_wait_us":3,"thread_start_us":14661,"threads_started":1}
I20260808 04:04:40.798797 15081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:40.765353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33570 (local address 127.5.83.62:39395)
0808 04:04:40.765552 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:40.765559 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:40.780650 (+ 15091us) server_negotiation.cc:407] Connection header received
0808 04:04:40.780813 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:40.780818 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:40.780904 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:40.780998 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:40.783136 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:40.784262 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:40.786278 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:40.786932 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:40.787047 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:40.791068 (+ 4021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:40.791075 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:40.791080 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:40.791118 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:40.795579 (+ 4461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:40.796416 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:40.796421 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:40.796425 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:40.796581 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:40.797095 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:40.797100 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:40.797103 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:40.797471 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:40.797611 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:40.798217 (+ 606us) server_negotiation.cc:299] Negotiation successful
0808 04:04:40.798453 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 04:04:40.800684 15080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:40.765630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59250 (local address 127.5.83.60:34729)
0808 04:04:40.765798 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:40.765803 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:40.779913 (+ 14110us) server_negotiation.cc:407] Connection header received
0808 04:04:40.780101 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:40.780105 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:40.780174 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:40.780259 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:40.787426 (+ 7167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:40.788557 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:40.790531 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:40.791284 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:40.791383 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:40.794093 (+ 2710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:40.794099 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:40.794102 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:40.794135 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:40.797892 (+ 3757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:40.798622 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:40.798628 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:40.798632 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:40.798762 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:40.799126 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:40.799131 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:40.799134 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:40.799534 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:40.799681 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:40.800095 (+ 414us) server_negotiation.cc:299] Negotiation successful
0808 04:04:40.800311 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":65}
I20260808 04:04:40.800478 15109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:40.772251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:39395 (local address 127.0.0.1:33570)
0808 04:04:40.780547 (+ 8296us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:40.780571 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:40.780692 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:40.781032 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:40.781036 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:40.782664 (+ 1628us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:04:40.782949 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:40.782962 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:40.784465 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:40.784471 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:40.786110 (+ 1639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:40.786125 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:40.787377 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:40.787385 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:40.787657 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:40.789583 (+ 1926us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:04:40.789605 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:04:40.790850 (+ 1245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:04:40.795763 (+ 4913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:40.795767 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:40.795771 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:40.796270 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:40.796735 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:40.796739 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:40.796743 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:40.796946 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:40.797699 (+ 753us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:04:40.797703 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:04:40.797920 (+ 217us) client_negotiation.cc:769] Sending connection context
0808 04:04:40.799299 (+ 1379us) client_negotiation.cc:240] Negotiation successful
0808 04:04:40.800015 (+ 716us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8174,"mutex_wait_us":2,"thread_start_us":8510,"threads_started":1}
I20260808 04:04:40.801211 15107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:40.765783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.60:34729 (local address 127.0.0.1:59250)
0808 04:04:40.779807 (+ 14024us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:40.779842 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:40.779978 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:40.786318 (+ 6340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:40.786322 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:40.786942 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:04:40.787258 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:40.787270 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:40.788800 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:40.788805 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:40.790368 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:40.790382 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:40.791655 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:40.791661 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:40.791865 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:40.792464 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:04:40.792486 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:04:40.793916 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:04:40.798029 (+ 4113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:40.798033 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:40.798037 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:40.798510 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:40.798894 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:40.798898 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:40.798906 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:40.799034 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:40.799653 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:04:40.799658 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:04:40.799857 (+ 199us) client_negotiation.cc:769] Sending connection context
0808 04:04:40.800511 (+ 654us) client_negotiation.cc:240] Negotiation successful
0808 04:04:40.800781 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13675,"thread_start_us":440,"threads_started":1}
Aug 08 04:04:40 dist-test-slave-1l3l krb5kdc[13966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161877, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.83.62@KRBTEST.COM for oryx/127.5.83.60@KRBTEST.COM
I20260808 04:04:40.822094 15010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:40.806452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.60:34729 (local address 127.0.0.1:59266)
0808 04:04:40.806643 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:40.806661 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:40.806765 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:40.807179 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:40.807183 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:40.807637 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:04:40.807895 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:40.807908 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:40.809321 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:40.809327 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:40.810746 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:40.810762 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:40.811913 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:40.811919 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:40.812112 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:40.812712 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:04:40.812733 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:04:40.815338 (+ 2605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:04:40.819285 (+ 3947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:40.819291 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:40.819295 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:40.819859 (+ 564us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:40.820250 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:40.820256 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:40.820260 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:40.820419 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:40.821027 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:04:40.821031 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:04:40.821273 (+ 242us) client_negotiation.cc:769] Sending connection context
0808 04:04:40.821450 (+ 177us) client_negotiation.cc:240] Negotiation successful
0808 04:04:40.821724 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260808 04:04:40.822176 15080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:40.806538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59266 (local address 127.5.83.60:34729)
0808 04:04:40.806694 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:40.806700 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:40.806721 (+ 21us) server_negotiation.cc:407] Connection header received
0808 04:04:40.806928 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:40.806934 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:40.807038 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:40.807137 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:40.808069 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:40.809142 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:40.810904 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:40.811748 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:40.811879 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:40.815543 (+ 3664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:40.815550 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:40.815555 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:40.815594 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:40.819111 (+ 3517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:40.819977 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:40.819983 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:40.819987 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:40.820150 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:40.820554 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:40.820559 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:40.820563 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:40.820903 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:40.821042 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:40.821545 (+ 503us) server_negotiation.cc:299] Negotiation successful
0808 04:04:40.821755 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 04:04:40.824029 14029 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.5.83.60:34729: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "f0960abfe8684bd79c949f4811b75825"
member_type: NON_VOTER
last_known_addr {
host: "127.5.83.60"
port: 34729
}
attrs {
promote: true
}
}
dest_uuid: "b254403e06ed4a0fbad7433dbbb9f425"
cas_config_opid_index: 5
I20260808 04:04:40.825104 14029 consensus_queue.cc:237] T 00000000000000000000000000000000 P b254403e06ed4a0fbad7433dbbb9f425 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b254403e06ed4a0fbad7433dbbb9f425" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 39395 } } peers { permanent_uuid: "72fecab4e79d4b7baa3fb285de73d640" member_type: NON_VOTER last_known_addr { host: "127.5.83.61" port: 41591 } attrs { promote: true } } peers { permanent_uuid: "f0960abfe8684bd79c949f4811b75825" member_type: NON_VOTER last_known_addr { host: "127.5.83.60" port: 34729 } attrs { promote: true } }
I20260808 04:04:40.826887 15009 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b254403e06ed4a0fbad7433dbbb9f425 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER f0960abfe8684bd79c949f4811b75825 (127.5.83.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b254403e06ed4a0fbad7433dbbb9f425" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 39395 } } peers { permanent_uuid: "72fecab4e79d4b7baa3fb285de73d640" member_type: NON_VOTER last_known_addr { host: "127.5.83.61" port: 41591 } attrs { promote: true } } peers { permanent_uuid: "f0960abfe8684bd79c949f4811b75825" member_type: NON_VOTER last_known_addr { host: "127.5.83.60" port: 34729 } attrs { promote: true } } }
I20260808 04:04:40.827538 14925 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 72fecab4e79d4b7baa3fb285de73d640 [term 1 LEARNER]: Refusing update from remote peer b254403e06ed4a0fbad7433dbbb9f425: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260808 04:04:40.830796 15048 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0960abfe8684bd79c949f4811b75825 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:04:40.834873 15048 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f0960abfe8684bd79c949f4811b75825 [term 1 FOLLOWER]: Refusing update from remote peer b254403e06ed4a0fbad7433dbbb9f425: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260808 04:04:40.839131 15113 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b254403e06ed4a0fbad7433dbbb9f425 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72fecab4e79d4b7baa3fb285de73d640" member_type: NON_VOTER last_known_addr { host: "127.5.83.61" port: 41591 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260808 04:04:40.843952 15009 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.5.83.60:34729
I20260808 04:04:40.844621 14860 master.cc:537] Master@127.5.83.60:34729 shutting down...
I20260808 04:04:40.845819 15116 sys_catalog.cc:455] T 00000000000000000000000000000000 P b254403e06ed4a0fbad7433dbbb9f425 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b254403e06ed4a0fbad7433dbbb9f425" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b254403e06ed4a0fbad7433dbbb9f425" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 39395 } } peers { permanent_uuid: "72fecab4e79d4b7baa3fb285de73d640" member_type: NON_VOTER last_known_addr { host: "127.5.83.61" port: 41591 } attrs { promote: true } } peers { permanent_uuid: "f0960abfe8684bd79c949f4811b75825" member_type: NON_VOTER last_known_addr { host: "127.5.83.60" port: 34729 } attrs { promote: true } } }
I20260808 04:04:40.846323 15116 sys_catalog.cc:458] T 00000000000000000000000000000000 P b254403e06ed4a0fbad7433dbbb9f425 [sys.catalog]: This master's current role is: LEADER
I20260808 04:04:40.846673 15113 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b254403e06ed4a0fbad7433dbbb9f425 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0960abfe8684bd79c949f4811b75825" member_type: NON_VOTER last_known_addr { host: "127.5.83.60" port: 34729 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:04:40.854885 14925 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 72fecab4e79d4b7baa3fb285de73d640 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER f0960abfe8684bd79c949f4811b75825 (127.5.83.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b254403e06ed4a0fbad7433dbbb9f425" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 39395 } } peers { permanent_uuid: "72fecab4e79d4b7baa3fb285de73d640" member_type: NON_VOTER last_known_addr { host: "127.5.83.61" port: 41591 } attrs { promote: true } } peers { permanent_uuid: "f0960abfe8684bd79c949f4811b75825" member_type: NON_VOTER last_known_addr { host: "127.5.83.60" port: 34729 } attrs { promote: true } } }
I20260808 04:04:40.858896 14969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72fecab4e79d4b7baa3fb285de73d640 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b254403e06ed4a0fbad7433dbbb9f425" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b254403e06ed4a0fbad7433dbbb9f425" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 39395 } } peers { permanent_uuid: "72fecab4e79d4b7baa3fb285de73d640" member_type: NON_VOTER last_known_addr { host: "127.5.83.61" port: 41591 } attrs { promote: true } } peers { permanent_uuid: "f0960abfe8684bd79c949f4811b75825" member_type: NON_VOTER last_known_addr { host: "127.5.83.60" port: 34729 } attrs { promote: true } } }
I20260808 04:04:40.859131 14969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72fecab4e79d4b7baa3fb285de73d640 [sys.catalog]: This master's current role is: LEARNER
I20260808 04:04:40.860073 15113 raft_consensus.cc:1050] T 00000000000000000000000000000000 P b254403e06ed4a0fbad7433dbbb9f425: attempting to promote NON_VOTER 72fecab4e79d4b7baa3fb285de73d640 to VOTER
I20260808 04:04:40.860746 15113 consensus_queue.cc:237] T 00000000000000000000000000000000 P b254403e06ed4a0fbad7433dbbb9f425 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b254403e06ed4a0fbad7433dbbb9f425" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 39395 } } peers { permanent_uuid: "72fecab4e79d4b7baa3fb285de73d640" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 41591 } attrs { promote: false } } peers { permanent_uuid: "f0960abfe8684bd79c949f4811b75825" member_type: NON_VOTER last_known_addr { host: "127.5.83.60" port: 34729 } attrs { promote: true } }
I20260808 04:04:40.862720 14925 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 72fecab4e79d4b7baa3fb285de73d640 [term 1 LEARNER]: Refusing update from remote peer b254403e06ed4a0fbad7433dbbb9f425: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260808 04:04:40.863191 15113 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b254403e06ed4a0fbad7433dbbb9f425 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72fecab4e79d4b7baa3fb285de73d640" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 41591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260808 04:04:40.865144 15009 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b254403e06ed4a0fbad7433dbbb9f425 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 72fecab4e79d4b7baa3fb285de73d640 (127.5.83.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b254403e06ed4a0fbad7433dbbb9f425" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 39395 } } peers { permanent_uuid: "72fecab4e79d4b7baa3fb285de73d640" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 41591 } attrs { promote: false } } peers { permanent_uuid: "f0960abfe8684bd79c949f4811b75825" member_type: NON_VOTER last_known_addr { host: "127.5.83.60" port: 34729 } attrs { promote: true } } }
I20260808 04:04:40.865823 14925 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 72fecab4e79d4b7baa3fb285de73d640 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 72fecab4e79d4b7baa3fb285de73d640 (127.5.83.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b254403e06ed4a0fbad7433dbbb9f425" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 39395 } } peers { permanent_uuid: "72fecab4e79d4b7baa3fb285de73d640" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 41591 } attrs { promote: false } } peers { permanent_uuid: "f0960abfe8684bd79c949f4811b75825" member_type: NON_VOTER last_known_addr { host: "127.5.83.60" port: 34729 } attrs { promote: true } } }
I20260808 04:04:40.869227 15116 sys_catalog.cc:455] T 00000000000000000000000000000000 P b254403e06ed4a0fbad7433dbbb9f425 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b254403e06ed4a0fbad7433dbbb9f425" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b254403e06ed4a0fbad7433dbbb9f425" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 39395 } } peers { permanent_uuid: "72fecab4e79d4b7baa3fb285de73d640" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 41591 } attrs { promote: false } } peers { permanent_uuid: "f0960abfe8684bd79c949f4811b75825" member_type: NON_VOTER last_known_addr { host: "127.5.83.60" port: 34729 } attrs { promote: true } } }
I20260808 04:04:40.869498 15116 sys_catalog.cc:458] T 00000000000000000000000000000000 P b254403e06ed4a0fbad7433dbbb9f425 [sys.catalog]: This master's current role is: LEADER
I20260808 04:04:40.869728 15113 sys_catalog.cc:455] T 00000000000000000000000000000000 P b254403e06ed4a0fbad7433dbbb9f425 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b254403e06ed4a0fbad7433dbbb9f425" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b254403e06ed4a0fbad7433dbbb9f425" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 39395 } } peers { permanent_uuid: "72fecab4e79d4b7baa3fb285de73d640" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 41591 } attrs { promote: false } } peers { permanent_uuid: "f0960abfe8684bd79c949f4811b75825" member_type: NON_VOTER last_known_addr { host: "127.5.83.60" port: 34729 } attrs { promote: true } } }
I20260808 04:04:40.870041 15113 sys_catalog.cc:458] T 00000000000000000000000000000000 P b254403e06ed4a0fbad7433dbbb9f425 [sys.catalog]: This master's current role is: LEADER
I20260808 04:04:40.870885 14969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72fecab4e79d4b7baa3fb285de73d640 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b254403e06ed4a0fbad7433dbbb9f425" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b254403e06ed4a0fbad7433dbbb9f425" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 39395 } } peers { permanent_uuid: "72fecab4e79d4b7baa3fb285de73d640" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 41591 } attrs { promote: false } } peers { permanent_uuid: "f0960abfe8684bd79c949f4811b75825" member_type: NON_VOTER last_known_addr { host: "127.5.83.60" port: 34729 } attrs { promote: true } } }
I20260808 04:04:40.871106 14969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72fecab4e79d4b7baa3fb285de73d640 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:04:40.886528 14860 raft_consensus.cc:2229] T 00000000000000000000000000000000 P f0960abfe8684bd79c949f4811b75825 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:04:40.886991 14860 raft_consensus.cc:2258] T 00000000000000000000000000000000 P f0960abfe8684bd79c949f4811b75825 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:04:40.887203 14860 tablet_replica.cc:330] T 00000000000000000000000000000000 P f0960abfe8684bd79c949f4811b75825: stopping tablet replica
W20260808 04:04:40.892005 14860 connection.cc:441] server connection from 127.0.0.1:59266 torn down before Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59266 (request call id 4) could send its response
W20260808 04:04:40.892303 14011 consensus_peers.cc:487] T 00000000000000000000000000000000 P b254403e06ed4a0fbad7433dbbb9f425 -> Peer f0960abfe8684bd79c949f4811b75825 (127.5.83.60:34729): Couldn't send request to peer f0960abfe8684bd79c949f4811b75825. Status: Network error: recv got EOF from unknown peer (error 108). This is attempt 1: this message will repeat every 5th retry.
I20260808 04:04:40.904254 14860 master.cc:552] Master@127.5.83.60:34729 shutdown complete.
I20260808 04:04:40.904357 14860 master_runner.cc:305] Clearing existing system tablet
I20260808 04:04:40.904817 14860 fs_manager.cc:359] Metadata directory not provided
I20260808 04:04:40.904918 14860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:04:40.915093 14860 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.006s sys 0.001s
I20260808 04:04:40.915524 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 14121
I20260808 04:04:40.932541 15123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:40.933053 14860 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:04:40.933424 14860 fs_manager.cc:725] Time spent opening block manager: real 0.017s user 0.002s sys 0.014s
I20260808 04:04:40.933526 14860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/master-2/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/master-2/wal
uuid: "f0960abfe8684bd79c949f4811b75825"
format_stamp: "Formatted at 2026-08-08 04:04:40 on dist-test-slave-1l3l"
server_key: "a68b794425ead5f714388c4a5ac41712"
server_key_iv: "4523951aaf975e745cc19ada88f527af"
server_key_version: "encryptionkey@0"
I20260808 04:04:40.938725 14860 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P f0960abfe8684bd79c949f4811b75825: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:04:40.945624 14860 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P f0960abfe8684bd79c949f4811b75825: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260808 04:04:40.945806 14860 log.cc:1196] T 00000000000000000000000000000000 P f0960abfe8684bd79c949f4811b75825: Deleting WAL directory at /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260808 04:04:40.946310 14860 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P f0960abfe8684bd79c949f4811b75825: Deleting consensus metadata
I20260808 04:04:40.946615 14860 master_runner.cc:315] Copying system tablet from 127.5.83.62:39395
I20260808 04:04:40.947799 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.1:45257
--local_ip_for_outbound_sockets=127.5.83.1
--tserver_master_addrs=127.5.83.62:39395
--webserver_port=38921
--webserver_interface=127.5.83.1
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/krb5kdc/oryx_127.5.83.1.keytab
--principal=oryx/127.5.83.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:46157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.5.83.62:39395,127.5.83.61:41591
--tserver_master_addrs=127.5.83.62:39395,127.5.83.61:41591,127.5.83.60:34729 with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:04:40.960981 14860 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P f0960abfe8684bd79c949f4811b75825: tablet copy: Beginning tablet copy session from remote peer at address 127.5.83.62:39395
I20260808 04:04:40.982342 15081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:40.961872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33584 (local address 127.5.83.62:39395)
0808 04:04:40.962129 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:40.962135 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:40.967889 (+ 5754us) server_negotiation.cc:407] Connection header received
0808 04:04:40.968175 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:40.968182 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:40.968283 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:40.968403 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:40.969483 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:40.970717 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:40.972653 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:40.973363 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:40.973498 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:40.975864 (+ 2366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:40.975872 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:40.975877 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:40.975918 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:40.979345 (+ 3427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:40.980137 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:40.980143 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:40.980147 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:40.980304 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:40.980730 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:40.980735 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:40.980739 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:40.981057 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:40.981203 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:40.981769 (+ 566us) server_negotiation.cc:299] Negotiation successful
0808 04:04:40.981997 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
I20260808 04:04:40.982395 15132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:40.961861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:39395 (local address 127.0.0.1:33584)
0808 04:04:40.967763 (+ 5902us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:40.967785 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:40.968026 (+ 241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:40.968451 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:40.968456 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:40.969016 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:04:40.969314 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:40.969328 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:40.970877 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:40.970883 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:40.972501 (+ 1618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:40.972518 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:40.973559 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:40.973565 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:40.973751 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:40.974480 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:04:40.974503 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:04:40.975640 (+ 1137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:04:40.979482 (+ 3842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:40.979488 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:40.979493 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:40.980009 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:40.980428 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:40.980433 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:40.980437 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:40.980613 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:40.981266 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:04:40.981271 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:04:40.981469 (+ 198us) client_negotiation.cc:769] Sending connection context
0808 04:04:40.981619 (+ 150us) client_negotiation.cc:240] Negotiation successful
0808 04:04:40.981904 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5729,"thread_start_us":5580,"threads_started":1}
I20260808 04:04:40.983300 14066 tablet_copy_service.cc:140] P b254403e06ed4a0fbad7433dbbb9f425: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer f0960abfe8684bd79c949f4811b75825 ({username='oryx', principal='oryx/127.5.83.60@KRBTEST.COM'} at 127.0.0.1:33584)
I20260808 04:04:40.983480 14066 tablet_copy_service.cc:161] P b254403e06ed4a0fbad7433dbbb9f425: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer f0960abfe8684bd79c949f4811b75825 at {username='oryx', principal='oryx/127.5.83.60@KRBTEST.COM'} at 127.0.0.1:33584: session id = f0960abfe8684bd79c949f4811b75825-00000000000000000000000000000000
I20260808 04:04:40.987046 14066 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P b254403e06ed4a0fbad7433dbbb9f425: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 04:04:40.988909 14860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260808 04:04:40.995541 14860 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P f0960abfe8684bd79c949f4811b75825: tablet copy: Starting download of 0 data blocks...
I20260808 04:04:40.995882 14860 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P f0960abfe8684bd79c949f4811b75825: tablet copy: Starting download of 1 WAL segments...
I20260808 04:04:40.998369 14860 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P f0960abfe8684bd79c949f4811b75825: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 04:04:41.002542 14066 tablet_copy_service.cc:342] P b254403e06ed4a0fbad7433dbbb9f425: Request end of tablet copy session f0960abfe8684bd79c949f4811b75825-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.5.83.60@KRBTEST.COM'} at 127.0.0.1:33584
I20260808 04:04:41.002694 14066 tablet_copy_service.cc:433] P b254403e06ed4a0fbad7433dbbb9f425: ending tablet copy session f0960abfe8684bd79c949f4811b75825-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer f0960abfe8684bd79c949f4811b75825
I20260808 04:04:41.013222 14860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:04:41.017262 15134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:41.025040 15135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:41.038190 15139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:04:41.038493 14860 server_base.cc:1034] running on GCE node
I20260808 04:04:41.039047 14860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:04:41.039772 14860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:04:41.040998 14860 hybrid_clock.cc:648] HybridClock initialized: now 1786161881040939 us; error 38 us; skew 500 ppm
I20260808 04:04:41.041214 14860 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:04:41 dist-test-slave-1l3l krb5kdc[13966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161881, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.83.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:04:41.043491 14860 init.cc:377] Logged in from keytab as oryx/127.5.83.60@KRBTEST.COM (short username oryx)
I20260808 04:04:41.052011 14860 webserver.cc:460] Webserver started at http://127.5.83.60:45787/ using document root <none> and password file <none>
I20260808 04:04:41.052337 14860 fs_manager.cc:359] Metadata directory not provided
I20260808 04:04:41.052413 14860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:04:41.055366 14860 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260808 04:04:41.057214 15147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:41.057574 14860 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 04:04:41.057741 14860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/master-2/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/master-2/wal
uuid: "f0960abfe8684bd79c949f4811b75825"
format_stamp: "Formatted at 2026-08-08 04:04:40 on dist-test-slave-1l3l"
server_key: "a68b794425ead5f714388c4a5ac41712"
server_key_iv: "4523951aaf975e745cc19ada88f527af"
server_key_version: "encryptionkey@0"
I20260808 04:04:41.057953 14860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:04:41.091876 14860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:04:41.096230 14860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:04:41.120735 14860 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.60:34729
I20260808 04:04:41.120807 15226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.60:34729 every 8 connection(s)
I20260808 04:04:41.122023 14860 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/master-2/data/info.pb
I20260808 04:04:41.127579 15227 sys_catalog.cc:263] Verifying existing consensus state
I20260808 04:04:41.131458 15227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0960abfe8684bd79c949f4811b75825: Bootstrap starting.
I20260808 04:04:41.150923 15227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0960abfe8684bd79c949f4811b75825: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 04:04:41.151467 15227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0960abfe8684bd79c949f4811b75825: Bootstrap complete.
I20260808 04:04:41.152652 15227 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0960abfe8684bd79c949f4811b75825 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b254403e06ed4a0fbad7433dbbb9f425" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 39395 } } peers { permanent_uuid: "72fecab4e79d4b7baa3fb285de73d640" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 41591 } attrs { promote: false } } peers { permanent_uuid: "f0960abfe8684bd79c949f4811b75825" member_type: NON_VOTER last_known_addr { host: "127.5.83.60" port: 34729 } attrs { promote: true } }
I20260808 04:04:41.152966 15227 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0960abfe8684bd79c949f4811b75825 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f0960abfe8684bd79c949f4811b75825, State: Initialized, Role: LEARNER
I20260808 04:04:41.153244 15227 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0960abfe8684bd79c949f4811b75825 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b254403e06ed4a0fbad7433dbbb9f425" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 39395 } } peers { permanent_uuid: "72fecab4e79d4b7baa3fb285de73d640" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 41591 } attrs { promote: false } } peers { permanent_uuid: "f0960abfe8684bd79c949f4811b75825" member_type: NON_VOTER last_known_addr { host: "127.5.83.60" port: 34729 } attrs { promote: true } }
I20260808 04:04:41.154091 15230 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0960abfe8684bd79c949f4811b75825 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b254403e06ed4a0fbad7433dbbb9f425" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 39395 } } peers { permanent_uuid: "72fecab4e79d4b7baa3fb285de73d640" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 41591 } attrs { promote: false } } peers { permanent_uuid: "f0960abfe8684bd79c949f4811b75825" member_type: NON_VOTER last_known_addr { host: "127.5.83.60" port: 34729 } attrs { promote: true } } }
I20260808 04:04:41.154392 15227 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0960abfe8684bd79c949f4811b75825 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:04:41.154552 15230 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0960abfe8684bd79c949f4811b75825 [sys.catalog]: This master's current role is: LEARNER
I20260808 04:04:41.170075 15244 catalog_manager.cc:1259] Loaded cluster ID: a15e7f4b3b484d4dbf31a2eea6d58df7
I20260808 04:04:41.170186 15244 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f0960abfe8684bd79c949f4811b75825: loading cluster ID for follower catalog manager: success
I20260808 04:04:41.173897 15244 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f0960abfe8684bd79c949f4811b75825: acquiring CA information for follower catalog manager: success
I20260808 04:04:41.176587 15244 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f0960abfe8684bd79c949f4811b75825: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260808 04:04:41.330933 15127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:04:41.331187 15127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:04:41.331321 15127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:04:41.339085 15127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:04:41.339186 15127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:04:41.339303 15127 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.1
I20260808 04:04:41.350921 15127 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:46157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/krb5kdc/oryx_127.5.83.1.keytab
--principal=oryx/127.5.83.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.1:45257
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.83.1
--webserver_port=38921
--tserver_master_addrs=127.5.83.62:39395,127.5.83.61:41591,127.5.83.60:34729
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.1
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:04:41.352329 15127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:04:41.353734 15127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:04:41.367239 15253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:41.367350 15255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:41.367568 15258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:04:41.368422 15127 server_base.cc:1034] running on GCE node
I20260808 04:04:41.368983 15127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:04:41.369943 15127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:04:41.371153 15127 hybrid_clock.cc:648] HybridClock initialized: now 1786161881371103 us; error 49 us; skew 500 ppm
I20260808 04:04:41.371423 15127 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:04:41 dist-test-slave-1l3l krb5kdc[13966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161881, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.83.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:04:41.374955 15127 init.cc:377] Logged in from keytab as oryx/127.5.83.1@KRBTEST.COM (short username oryx)
I20260808 04:04:41.381422 15127 webserver.cc:460] Webserver started at http://127.5.83.1:38921/ using document root <none> and password file <none>
I20260808 04:04:41.382283 15127 fs_manager.cc:359] Metadata directory not provided
I20260808 04:04:41.382441 15127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:04:41.386449 15263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:41.373925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.60:34729 (local address 127.0.0.1:59268)
0808 04:04:41.380099 (+ 6174us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:41.380127 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:41.380285 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:41.381013 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:41.381017 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:41.381044 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:04:41.381276 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:41.381290 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:41.383064 (+ 1774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:41.383068 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:41.384300 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:41.384320 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:41.385801 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:41.385805 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:41.385952 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:41.385987 (+ 35us) client_negotiation.cc:769] Sending connection context
0808 04:04:41.386064 (+ 77us) client_negotiation.cc:240] Negotiation successful
0808 04:04:41.386167 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6023,"thread_start_us":5929,"threads_started":1}
I20260808 04:04:41.386574 15264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:41.374022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59268 (local address 127.5.83.60:34729)
0808 04:04:41.380638 (+ 6616us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:41.380645 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:41.380665 (+ 20us) server_negotiation.cc:407] Connection header received
0808 04:04:41.380746 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:41.380751 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:41.380840 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:41.380979 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:04:41.381499 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:41.382917 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:41.384454 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:41.385612 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:41.385745 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:41.386004 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:41.386110 (+ 106us) server_negotiation.cc:299] Negotiation successful
0808 04:04:41.386243 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6488,"thread_start_us":6379,"threads_started":1}
I20260808 04:04:41.387454 15189 raft_consensus.cc:1203] T 00000000000000000000000000000000 P f0960abfe8684bd79c949f4811b75825 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260808 04:04:41.388054 15230 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0960abfe8684bd79c949f4811b75825 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b254403e06ed4a0fbad7433dbbb9f425. Latest consensus state: current_term: 1 leader_uuid: "b254403e06ed4a0fbad7433dbbb9f425" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b254403e06ed4a0fbad7433dbbb9f425" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 39395 } } peers { permanent_uuid: "72fecab4e79d4b7baa3fb285de73d640" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 41591 } attrs { promote: false } } peers { permanent_uuid: "f0960abfe8684bd79c949f4811b75825" member_type: NON_VOTER last_known_addr { host: "127.5.83.60" port: 34729 } attrs { promote: true } } }
I20260808 04:04:41.388331 15230 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0960abfe8684bd79c949f4811b75825 [sys.catalog]: This master's current role is: LEARNER
I20260808 04:04:41.389009 15127 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 04:04:41.391685 15270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:41.393042 15127 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 04:04:41.393170 15127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-0/wal
uuid: "82a19f4b641448c384fdc9198ab1143e"
format_stamp: "Formatted at 2026-08-08 04:04:38 on dist-test-slave-1l3l"
server_key: "ebba60576942b7c95247b2228dc30bbe"
server_key_iv: "f58da4ec03823b6b87d392522e07330a"
server_key_version: "encryptionkey@0"
I20260808 04:04:41.394083 15127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:04:41.419855 15127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:04:41.424050 15127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:04:41.424296 15127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:04:41.425315 15127 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:04:41.426671 15127 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:04:41.426759 15127 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:41.426841 15127 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:04:41.426894 15127 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:41.459760 15127 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.1:45257
I20260808 04:04:41.459844 15431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.1:45257 every 8 connection(s)
I20260808 04:04:41.461155 15127 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
I20260808 04:04:41.465407 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 15127
I20260808 04:04:41.465766 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 14308
Aug 08 04:04:41 dist-test-slave-1l3l krb5kdc[13966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161881, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.83.1@KRBTEST.COM for oryx/127.5.83.61@KRBTEST.COM
Aug 08 04:04:41 dist-test-slave-1l3l krb5kdc[13966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.83.2@KRBTEST.COM
Aug 08 04:04:41 dist-test-slave-1l3l krb5kdc[13966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161881, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.83.1@KRBTEST.COM for oryx/127.5.83.60@KRBTEST.COM
Aug 08 04:04:41 dist-test-slave-1l3l krb5kdc[13966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161881, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.83.1@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM
I20260808 04:04:41.490582 15264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:41.464055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.1:46767 (local address 127.5.83.60:34729)
0808 04:04:41.464404 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:41.464410 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:41.468000 (+ 3590us) server_negotiation.cc:407] Connection header received
0808 04:04:41.469877 (+ 1877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:41.469883 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:41.469960 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:41.470054 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:41.472574 (+ 2520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:41.473664 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:41.476412 (+ 2748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:41.477047 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:41.477148 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:41.483793 (+ 6645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:41.483800 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:41.483804 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:41.483844 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:41.487003 (+ 3159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:41.487899 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:41.487905 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:41.487908 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:41.488063 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:41.488632 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:41.488636 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:41.488639 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:41.488977 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:41.489103 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:41.490018 (+ 915us) server_negotiation.cc:299] Negotiation successful
0808 04:04:41.490250 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230}
I20260808 04:04:41.490645 15081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:41.464156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.1:43941 (local address 127.5.83.62:39395)
0808 04:04:41.464358 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:41.464364 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:41.467740 (+ 3376us) server_negotiation.cc:407] Connection header received
0808 04:04:41.469784 (+ 2044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:41.469790 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:41.469875 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:41.469946 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:41.472559 (+ 2613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:41.473578 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:41.476430 (+ 2852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:41.477054 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:41.477151 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:41.484103 (+ 6952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:41.484110 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:41.484115 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:41.484153 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:41.487587 (+ 3434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:41.488519 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:41.488524 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:41.488527 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:41.488660 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:41.489097 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:41.489102 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:41.489105 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:41.489474 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:41.489593 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:41.490121 (+ 528us) server_negotiation.cc:299] Negotiation successful
0808 04:04:41.490334 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 04:04:41.490721 15438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:41.463856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.1:59065 (local address 127.5.83.61:41591)
0808 04:04:41.464653 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:41.464658 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:41.468069 (+ 3411us) server_negotiation.cc:407] Connection header received
0808 04:04:41.469862 (+ 1793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:41.469868 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:41.469960 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:41.470077 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:41.472575 (+ 2498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:41.473676 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:41.476412 (+ 2736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:41.477180 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:41.477265 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:41.482198 (+ 4933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:41.482205 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:41.482210 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:41.482249 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:41.487763 (+ 5514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:41.488643 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:41.488648 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:41.488651 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:41.488783 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:41.489227 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:41.489232 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:41.489235 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:41.489553 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:41.489680 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:41.490181 (+ 501us) server_negotiation.cc:299] Negotiation successful
0808 04:04:41.490372 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":703,"thread_start_us":601,"threads_started":1}
I20260808 04:04:41.491155 15446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:41.466698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42896 (local address 127.5.83.2:46681)
0808 04:04:41.468229 (+ 1531us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:41.468234 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:41.468712 (+ 478us) server_negotiation.cc:407] Connection header received
0808 04:04:41.468865 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:41.468872 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:41.469044 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:41.469174 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:41.471201 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:41.472732 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:41.474786 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:41.475594 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:41.475691 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:41.482590 (+ 6899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:41.482602 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:41.482612 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:41.482644 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:41.487228 (+ 4584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:41.488070 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:41.488078 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:41.488081 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:41.488235 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:41.488738 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:41.488743 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:41.488746 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:41.489180 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:41.489436 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:41.490009 (+ 573us) server_negotiation.cc:299] Negotiation successful
0808 04:04:41.490266 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1385,"thread_start_us":289,"threads_started":1}
I20260808 04:04:41.492766 15441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:41.465360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.60:34729 (local address 127.5.83.1:46767)
0808 04:04:41.467896 (+ 2536us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:41.467918 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:41.469483 (+ 1565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:41.470357 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:41.470368 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:41.471340 (+ 972us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:04:41.472375 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:41.472396 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:41.473860 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:41.473865 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:41.476255 (+ 2390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:41.476266 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:41.477532 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:41.477540 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:41.477873 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:41.478740 (+ 867us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:04:41.478766 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:04:41.483558 (+ 4792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:04:41.487165 (+ 3607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:41.487173 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:41.487187 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:41.487759 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:41.488224 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:41.488229 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:41.488232 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:41.488495 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:41.489188 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:04:41.489195 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:04:41.489572 (+ 377us) client_negotiation.cc:769] Sending connection context
0808 04:04:41.490849 (+ 1277us) client_negotiation.cc:240] Negotiation successful
0808 04:04:41.491738 (+ 889us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2420,"mutex_wait_us":807,"spinlock_wait_cycles":7296,"thread_start_us":1889,"threads_started":1}
I20260808 04:04:41.492807 15443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:41.464977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:39395 (local address 127.5.83.1:43941)
0808 04:04:41.467609 (+ 2632us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:41.467631 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:41.469483 (+ 1852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:41.470358 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:41.470368 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:41.471358 (+ 990us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:04:41.472365 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:41.472389 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:41.473785 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:41.473790 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:41.476256 (+ 2466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:41.476271 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:41.477506 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:41.477514 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:41.477888 (+ 374us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:41.478718 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:04:41.478751 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:04:41.483892 (+ 5141us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:04:41.487776 (+ 3884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:41.487781 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:41.487785 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:41.488364 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:41.488800 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:41.488805 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:41.488808 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:41.488963 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:41.489674 (+ 711us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:04:41.489678 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:04:41.489820 (+ 142us) client_negotiation.cc:769] Sending connection context
0808 04:04:41.490877 (+ 1057us) client_negotiation.cc:240] Negotiation successful
0808 04:04:41.491582 (+ 705us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2492,"mutex_wait_us":619,"spinlock_wait_cycles":105216,"thread_start_us":2562,"threads_started":1}
I20260808 04:04:41.493067 14498 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:04:41.494251 15434 heartbeater.cc:347] Connected to a master server at 127.5.83.62:39395
I20260808 04:04:41.494304 15432 heartbeater.cc:347] Connected to a master server at 127.5.83.60:34729
I20260808 04:04:41.494531 15434 heartbeater.cc:464] Registering TS with master...
I20260808 04:04:41.494537 15432 heartbeater.cc:464] Registering TS with master...
I20260808 04:04:41.495389 15434 heartbeater.cc:511] Master 127.5.83.62:39395 requested a full tablet report, sending...
I20260808 04:04:41.495919 15439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:41.464901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.61:41591 (local address 127.5.83.1:59065)
0808 04:04:41.467299 (+ 2398us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:41.467358 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:41.469477 (+ 2119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:41.470357 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:41.470368 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:41.471253 (+ 885us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:04:41.472373 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:41.472396 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:41.473865 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:41.473869 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:41.476245 (+ 2376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:41.476267 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:41.477551 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:41.477556 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:41.477903 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:41.478817 (+ 914us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:04:41.478839 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:04:41.481926 (+ 3087us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:04:41.487894 (+ 5968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:41.487898 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:41.487901 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:41.488524 (+ 623us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:41.488921 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:41.488930 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:41.488933 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:41.489105 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:41.489761 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:04:41.489765 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:04:41.489913 (+ 148us) client_negotiation.cc:769] Sending connection context
0808 04:04:41.490927 (+ 1014us) client_negotiation.cc:240] Negotiation successful
0808 04:04:41.491402 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1611,"mutex_wait_us":506,"spinlock_wait_cycles":3072,"thread_start_us":504,"threads_started":1}
I20260808 04:04:41.496112 15432 heartbeater.cc:511] Master 127.5.83.60:34729 requested a full tablet report, sending...
I20260808 04:04:41.497303 14029 ts_manager.cc:194] Re-registered known tserver with Master: 82a19f4b641448c384fdc9198ab1143e (127.5.83.1:45257)
I20260808 04:04:41.498322 14029 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.83.1@KRBTEST.COM'} at 127.5.83.1:43941
I20260808 04:04:41.498574 15172 ts_manager.cc:194] Registered new tserver with Master: 82a19f4b641448c384fdc9198ab1143e (127.5.83.1:45257)
I20260808 04:04:41.499320 15435 heartbeater.cc:347] Connected to a master server at 127.5.83.61:41591
I20260808 04:04:41.499454 15435 heartbeater.cc:464] Registering TS with master...
I20260808 04:04:41.500315 15435 heartbeater.cc:511] Master 127.5.83.61:41591 requested a full tablet report, sending...
I20260808 04:04:41.501663 14912 ts_manager.cc:194] Registered new tserver with Master: 82a19f4b641448c384fdc9198ab1143e (127.5.83.1:45257)
I20260808 04:04:41.581343 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 14308
I20260808 04:04:41.600302 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.2:46681
--local_ip_for_outbound_sockets=127.5.83.2
--tserver_master_addrs=127.5.83.62:39395
--webserver_port=41269
--webserver_interface=127.5.83.2
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/krb5kdc/oryx_127.5.83.2.keytab
--principal=oryx/127.5.83.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:46157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.5.83.62:39395,127.5.83.61:41591
--tserver_master_addrs=127.5.83.62:39395,127.5.83.61:41591,127.5.83.60:34729 with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:04:41.820901 15113 sys_catalog.cc:455] T 00000000000000000000000000000000 P b254403e06ed4a0fbad7433dbbb9f425 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b254403e06ed4a0fbad7433dbbb9f425" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b254403e06ed4a0fbad7433dbbb9f425" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 39395 } } peers { permanent_uuid: "72fecab4e79d4b7baa3fb285de73d640" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 41591 } attrs { promote: false } } peers { permanent_uuid: "f0960abfe8684bd79c949f4811b75825" member_type: NON_VOTER last_known_addr { host: "127.5.83.60" port: 34729 } attrs { promote: true } } }
I20260808 04:04:41.821297 15113 sys_catalog.cc:458] T 00000000000000000000000000000000 P b254403e06ed4a0fbad7433dbbb9f425 [sys.catalog]: This master's current role is: LEADER
I20260808 04:04:41.822804 15450 raft_consensus.cc:1050] T 00000000000000000000000000000000 P b254403e06ed4a0fbad7433dbbb9f425: attempting to promote NON_VOTER f0960abfe8684bd79c949f4811b75825 to VOTER
I20260808 04:04:41.823406 15450 consensus_queue.cc:237] T 00000000000000000000000000000000 P b254403e06ed4a0fbad7433dbbb9f425 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b254403e06ed4a0fbad7433dbbb9f425" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 39395 } } peers { permanent_uuid: "72fecab4e79d4b7baa3fb285de73d640" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 41591 } attrs { promote: false } } peers { permanent_uuid: "f0960abfe8684bd79c949f4811b75825" member_type: VOTER last_known_addr { host: "127.5.83.60" port: 34729 } attrs { promote: false } }
I20260808 04:04:41.825425 15189 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f0960abfe8684bd79c949f4811b75825 [term 1 LEARNER]: Refusing update from remote peer b254403e06ed4a0fbad7433dbbb9f425: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260808 04:04:41.825555 14925 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 72fecab4e79d4b7baa3fb285de73d640 [term 1 FOLLOWER]: Refusing update from remote peer b254403e06ed4a0fbad7433dbbb9f425: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260808 04:04:41.826037 15450 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b254403e06ed4a0fbad7433dbbb9f425 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0960abfe8684bd79c949f4811b75825" member_type: VOTER last_known_addr { host: "127.5.83.60" port: 34729 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260808 04:04:41.826424 15451 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b254403e06ed4a0fbad7433dbbb9f425 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72fecab4e79d4b7baa3fb285de73d640" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 41591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260808 04:04:41.829149 15451 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b254403e06ed4a0fbad7433dbbb9f425 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f0960abfe8684bd79c949f4811b75825 (127.5.83.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b254403e06ed4a0fbad7433dbbb9f425" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 39395 } } peers { permanent_uuid: "72fecab4e79d4b7baa3fb285de73d640" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 41591 } attrs { promote: false } } peers { permanent_uuid: "f0960abfe8684bd79c949f4811b75825" member_type: VOTER last_known_addr { host: "127.5.83.60" port: 34729 } attrs { promote: false } } }
I20260808 04:04:41.829828 14925 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 72fecab4e79d4b7baa3fb285de73d640 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f0960abfe8684bd79c949f4811b75825 (127.5.83.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b254403e06ed4a0fbad7433dbbb9f425" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 39395 } } peers { permanent_uuid: "72fecab4e79d4b7baa3fb285de73d640" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 41591 } attrs { promote: false } } peers { permanent_uuid: "f0960abfe8684bd79c949f4811b75825" member_type: VOTER last_known_addr { host: "127.5.83.60" port: 34729 } attrs { promote: false } } }
I20260808 04:04:41.833184 15451 sys_catalog.cc:455] T 00000000000000000000000000000000 P b254403e06ed4a0fbad7433dbbb9f425 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b254403e06ed4a0fbad7433dbbb9f425" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b254403e06ed4a0fbad7433dbbb9f425" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 39395 } } peers { permanent_uuid: "72fecab4e79d4b7baa3fb285de73d640" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 41591 } attrs { promote: false } } peers { permanent_uuid: "f0960abfe8684bd79c949f4811b75825" member_type: VOTER last_known_addr { host: "127.5.83.60" port: 34729 } attrs { promote: false } } }
I20260808 04:04:41.833539 15451 sys_catalog.cc:458] T 00000000000000000000000000000000 P b254403e06ed4a0fbad7433dbbb9f425 [sys.catalog]: This master's current role is: LEADER
I20260808 04:04:41.833416 15113 sys_catalog.cc:455] T 00000000000000000000000000000000 P b254403e06ed4a0fbad7433dbbb9f425 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b254403e06ed4a0fbad7433dbbb9f425" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b254403e06ed4a0fbad7433dbbb9f425" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 39395 } } peers { permanent_uuid: "72fecab4e79d4b7baa3fb285de73d640" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 41591 } attrs { promote: false } } peers { permanent_uuid: "f0960abfe8684bd79c949f4811b75825" member_type: VOTER last_known_addr { host: "127.5.83.60" port: 34729 } attrs { promote: false } } }
I20260808 04:04:41.833747 15113 sys_catalog.cc:458] T 00000000000000000000000000000000 P b254403e06ed4a0fbad7433dbbb9f425 [sys.catalog]: This master's current role is: LEADER
I20260808 04:04:41.836165 15455 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72fecab4e79d4b7baa3fb285de73d640 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b254403e06ed4a0fbad7433dbbb9f425" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b254403e06ed4a0fbad7433dbbb9f425" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 39395 } } peers { permanent_uuid: "72fecab4e79d4b7baa3fb285de73d640" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 41591 } attrs { promote: false } } peers { permanent_uuid: "f0960abfe8684bd79c949f4811b75825" member_type: VOTER last_known_addr { host: "127.5.83.60" port: 34729 } attrs { promote: false } } }
I20260808 04:04:41.836613 15455 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72fecab4e79d4b7baa3fb285de73d640 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:04:41.842166 15113 sys_catalog.cc:455] T 00000000000000000000000000000000 P b254403e06ed4a0fbad7433dbbb9f425 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b254403e06ed4a0fbad7433dbbb9f425" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b254403e06ed4a0fbad7433dbbb9f425" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 39395 } } peers { permanent_uuid: "72fecab4e79d4b7baa3fb285de73d640" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 41591 } attrs { promote: false } } peers { permanent_uuid: "f0960abfe8684bd79c949f4811b75825" member_type: VOTER last_known_addr { host: "127.5.83.60" port: 34729 } attrs { promote: false } } }
I20260808 04:04:41.842440 15113 sys_catalog.cc:458] T 00000000000000000000000000000000 P b254403e06ed4a0fbad7433dbbb9f425 [sys.catalog]: This master's current role is: LEADER
I20260808 04:04:41.842900 15189 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f0960abfe8684bd79c949f4811b75825 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f0960abfe8684bd79c949f4811b75825 (127.5.83.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b254403e06ed4a0fbad7433dbbb9f425" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 39395 } } peers { permanent_uuid: "72fecab4e79d4b7baa3fb285de73d640" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 41591 } attrs { promote: false } } peers { permanent_uuid: "f0960abfe8684bd79c949f4811b75825" member_type: VOTER last_known_addr { host: "127.5.83.60" port: 34729 } attrs { promote: false } } }
I20260808 04:04:41.845896 15230 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0960abfe8684bd79c949f4811b75825 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b254403e06ed4a0fbad7433dbbb9f425" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b254403e06ed4a0fbad7433dbbb9f425" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 39395 } } peers { permanent_uuid: "72fecab4e79d4b7baa3fb285de73d640" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 41591 } attrs { promote: false } } peers { permanent_uuid: "f0960abfe8684bd79c949f4811b75825" member_type: VOTER last_known_addr { host: "127.5.83.60" port: 34729 } attrs { promote: false } } }
I20260808 04:04:41.846153 15230 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0960abfe8684bd79c949f4811b75825 [sys.catalog]: This master's current role is: FOLLOWER
W20260808 04:04:41.925439 15449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:04:41.925701 15449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:04:41.925794 15449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:04:41.932811 15449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:04:41.932910 15449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:04:41.933028 15449 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.2
I20260808 04:04:41.942296 15449 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:46157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/krb5kdc/oryx_127.5.83.2.keytab
--principal=oryx/127.5.83.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.2:46681
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.83.2
--webserver_port=41269
--tserver_master_addrs=127.5.83.62:39395,127.5.83.61:41591,127.5.83.60:34729
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.2
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:04:41.943444 15449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:04:41.944684 15449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:04:41.957669 15463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:41.957804 15468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:41.957895 15465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:04:41.958917 15449 server_base.cc:1034] running on GCE node
I20260808 04:04:41.959596 15449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:04:41.960606 15449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:04:41.961796 15449 hybrid_clock.cc:648] HybridClock initialized: now 1786161881961748 us; error 37 us; skew 500 ppm
I20260808 04:04:41.962122 15449 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:04:41 dist-test-slave-1l3l krb5kdc[13966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161881, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.83.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:04:41.965220 15449 init.cc:377] Logged in from keytab as oryx/127.5.83.2@KRBTEST.COM (short username oryx)
I20260808 04:04:41.966547 15449 webserver.cc:460] Webserver started at http://127.5.83.2:41269/ using document root <none> and password file <none>
I20260808 04:04:41.967192 15449 fs_manager.cc:359] Metadata directory not provided
I20260808 04:04:41.967321 15449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:04:41.972316 15449 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:04:41.974781 15479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:41.976025 15449 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 04:04:41.976166 15449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-1/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-1/wal
uuid: "861039c94e114a4c9afc2caeb3282448"
format_stamp: "Formatted at 2026-08-08 04:04:38 on dist-test-slave-1l3l"
server_key: "7cfc0a58299f7111a152a93b288496f7"
server_key_iv: "d12fc8583b9c11e6684cbfe1f7346a22"
server_key_version: "encryptionkey@0"
I20260808 04:04:41.977066 15449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:04:41.989652 15449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:04:41.993736 15449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:04:41.993988 15449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:04:41.994853 15449 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:04:41.996145 15449 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:04:41.996232 15449 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:41.996330 15449 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:04:41.996405 15449 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:42.027455 15449 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.2:46681
I20260808 04:04:42.027525 15645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.2:46681 every 8 connection(s)
I20260808 04:04:42.028801 15449 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
I20260808 04:04:42.032711 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 15449
I20260808 04:04:42.032984 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 14518
Aug 08 04:04:42 dist-test-slave-1l3l krb5kdc[13966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161881, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.83.2@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM
Aug 08 04:04:42 dist-test-slave-1l3l krb5kdc[13966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161881, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.83.2@KRBTEST.COM for oryx/127.5.83.61@KRBTEST.COM
Aug 08 04:04:42 dist-test-slave-1l3l krb5kdc[13966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161881, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.83.2@KRBTEST.COM for oryx/127.5.83.60@KRBTEST.COM
Aug 08 04:04:42 dist-test-slave-1l3l krb5kdc[13966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.83.3@KRBTEST.COM
I20260808 04:04:42.081689 15659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:42.031903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.2:47121 (local address 127.5.83.61:41591)
0808 04:04:42.059435 (+ 27532us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:42.059442 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:42.060528 (+ 1086us) server_negotiation.cc:407] Connection header received
0808 04:04:42.061564 (+ 1036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:42.061569 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:42.061662 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:42.061763 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:42.064097 (+ 2334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:42.065182 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:42.067814 (+ 2632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:42.068432 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:42.068533 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:42.073550 (+ 5017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:42.073554 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:42.073558 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:42.073607 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:42.077798 (+ 4191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:42.078840 (+ 1042us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:42.078846 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:42.078849 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:42.079004 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:42.079532 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:42.079538 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:42.079541 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:42.079926 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:42.080048 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:42.081105 (+ 1057us) server_negotiation.cc:299] Negotiation successful
0808 04:04:42.081331 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27420,"thread_start_us":27333,"threads_started":1}
I20260808 04:04:42.082063 15658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:42.031807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.2:38703 (local address 127.5.83.60:34729)
0808 04:04:42.059030 (+ 27223us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:42.059037 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:42.060421 (+ 1384us) server_negotiation.cc:407] Connection header received
0808 04:04:42.061809 (+ 1388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:42.061815 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:42.061893 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:42.061972 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:42.065383 (+ 3411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:42.066446 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:42.069333 (+ 2887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:42.070016 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:42.070135 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:42.074036 (+ 3901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:42.074043 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:42.074047 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:42.074088 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:42.078183 (+ 4095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:42.078935 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:42.078940 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:42.078943 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:42.079087 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:42.079532 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:42.079537 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:42.079541 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:42.079902 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:42.080048 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:42.081103 (+ 1055us) server_negotiation.cc:299] Negotiation successful
0808 04:04:42.081332 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27062,"thread_start_us":27211,"threads_started":1}
I20260808 04:04:42.082093 15666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:42.034197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50892 (local address 127.5.83.3:46513)
0808 04:04:42.057447 (+ 23250us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:42.057456 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:42.061020 (+ 3564us) server_negotiation.cc:407] Connection header received
0808 04:04:42.061199 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:42.061208 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:42.061463 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:42.061613 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:42.064042 (+ 2429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:42.065133 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:42.066968 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:42.067787 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:42.067869 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:42.075102 (+ 7233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:42.075114 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:42.075128 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:42.075168 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:42.078629 (+ 3461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:42.079447 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:42.079456 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:42.079459 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:42.079618 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:42.080116 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:42.080122 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:42.080126 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:42.080548 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:42.080817 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:42.081409 (+ 592us) server_negotiation.cc:299] Negotiation successful
0808 04:04:42.081668 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23061,"thread_start_us":22685,"threads_started":1}
I20260808 04:04:42.082756 15654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:42.031504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.2:53859 (local address 127.5.83.62:39395)
0808 04:04:42.059878 (+ 28374us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:42.059883 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:42.059901 (+ 18us) server_negotiation.cc:407] Connection header received
0808 04:04:42.061383 (+ 1482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:42.061389 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:42.061478 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:42.061576 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:42.063981 (+ 2405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:42.065124 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:42.067794 (+ 2670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:42.068463 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:42.068571 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:42.074950 (+ 6379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:42.074957 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:42.074962 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:42.075007 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:42.079197 (+ 4190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:42.080370 (+ 1173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:42.080375 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:42.080378 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:42.080562 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:42.081195 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:42.081200 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:42.081203 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:42.081554 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:42.081678 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:42.082188 (+ 510us) server_negotiation.cc:299] Negotiation successful
0808 04:04:42.082415 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28261,"thread_start_us":28382,"threads_started":1}
I20260808 04:04:42.084105 14699 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:04:42.085603 15665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:42.032269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:39395 (local address 127.5.83.2:53859)
0808 04:04:42.058469 (+ 26200us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:42.058527 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:42.061070 (+ 2543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:42.061851 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:42.061861 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:42.062793 (+ 932us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:04:42.063803 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:42.063835 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:42.065284 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:42.065289 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:42.067610 (+ 2321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:42.067638 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:42.068669 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:42.068677 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:42.068994 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:42.069725 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:04:42.069759 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:04:42.074245 (+ 4486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:04:42.079685 (+ 5440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:42.079690 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:42.079693 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:42.080234 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:42.080798 (+ 564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:42.080803 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:42.080806 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:42.081060 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:42.081758 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:04:42.081763 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:04:42.081916 (+ 153us) client_negotiation.cc:769] Sending connection context
0808 04:04:42.082938 (+ 1022us) client_negotiation.cc:240] Negotiation successful
0808 04:04:42.083232 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25948,"mutex_wait_us":313,"thread_start_us":25275,"threads_started":1}
I20260808 04:04:42.085626 15662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:42.032269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.60:34729 (local address 127.5.83.2:38703)
0808 04:04:42.060322 (+ 28053us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:42.060340 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:42.061079 (+ 739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:42.062186 (+ 1107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:42.062190 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:42.062983 (+ 793us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:04:42.063803 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:42.063833 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:42.066634 (+ 2801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:42.066639 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:42.069151 (+ 2512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:42.069163 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:42.070497 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:42.070504 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:42.070702 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:42.071264 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:04:42.071286 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:04:42.073815 (+ 2529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:04:42.078345 (+ 4530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:42.078350 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:42.078353 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:42.078815 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:42.079212 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:42.079217 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:42.079220 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:42.079400 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:42.080145 (+ 745us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:04:42.080152 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:04:42.080629 (+ 477us) client_negotiation.cc:769] Sending connection context
0808 04:04:42.082689 (+ 2060us) client_negotiation.cc:240] Negotiation successful
0808 04:04:42.083389 (+ 700us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25465,"mutex_wait_us":207,"spinlock_wait_cycles":15104,"thread_start_us":27697,"threads_started":1}
I20260808 04:04:42.085856 15660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:42.033609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.61:41591 (local address 127.5.83.2:47121)
0808 04:04:42.060410 (+ 26801us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:42.060430 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:42.061421 (+ 991us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:42.061953 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:42.061958 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:42.062882 (+ 924us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:04:42.063882 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:42.063896 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:42.065338 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:42.065348 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:42.067612 (+ 2264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:42.067639 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:42.068840 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:42.068846 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:42.069063 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:42.069849 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:04:42.069872 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:04:42.073296 (+ 3424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:04:42.077975 (+ 4679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:42.077984 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:42.078001 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:42.078678 (+ 677us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:42.079147 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:42.079151 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:42.079155 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:42.079400 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:42.080145 (+ 745us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:04:42.080152 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:04:42.080629 (+ 477us) client_negotiation.cc:769] Sending connection context
0808 04:04:42.081982 (+ 1353us) client_negotiation.cc:240] Negotiation successful
0808 04:04:42.082505 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26636,"mutex_wait_us":368,"spinlock_wait_cycles":44672,"thread_start_us":24623,"threads_started":1}
I20260808 04:04:42.087805 15647 heartbeater.cc:347] Connected to a master server at 127.5.83.62:39395
I20260808 04:04:42.087855 15648 heartbeater.cc:347] Connected to a master server at 127.5.83.61:41591
I20260808 04:04:42.087879 15646 heartbeater.cc:347] Connected to a master server at 127.5.83.60:34729
I20260808 04:04:42.088219 15646 heartbeater.cc:464] Registering TS with master...
I20260808 04:04:42.088244 15648 heartbeater.cc:464] Registering TS with master...
I20260808 04:04:42.088220 15647 heartbeater.cc:464] Registering TS with master...
I20260808 04:04:42.089475 15648 heartbeater.cc:511] Master 127.5.83.61:41591 requested a full tablet report, sending...
I20260808 04:04:42.089483 15647 heartbeater.cc:511] Master 127.5.83.62:39395 requested a full tablet report, sending...
I20260808 04:04:42.089529 15646 heartbeater.cc:511] Master 127.5.83.60:34729 requested a full tablet report, sending...
I20260808 04:04:42.091747 14029 ts_manager.cc:194] Re-registered known tserver with Master: 861039c94e114a4c9afc2caeb3282448 (127.5.83.2:46681)
I20260808 04:04:42.092437 15172 ts_manager.cc:194] Registered new tserver with Master: 861039c94e114a4c9afc2caeb3282448 (127.5.83.2:46681)
I20260808 04:04:42.093204 14029 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.83.2@KRBTEST.COM'} at 127.5.83.2:53859
I20260808 04:04:42.093366 14912 ts_manager.cc:194] Registered new tserver with Master: 861039c94e114a4c9afc2caeb3282448 (127.5.83.2:46681)
I20260808 04:04:42.173554 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 14518
I20260808 04:04:42.195116 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.3:46513
--local_ip_for_outbound_sockets=127.5.83.3
--tserver_master_addrs=127.5.83.62:39395
--webserver_port=39401
--webserver_interface=127.5.83.3
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/krb5kdc/oryx_127.5.83.3.keytab
--principal=oryx/127.5.83.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:46157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.5.83.62:39395,127.5.83.61:41591
--tserver_master_addrs=127.5.83.62:39395,127.5.83.61:41591,127.5.83.60:34729 with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:04:42.501022 15434 heartbeater.cc:503] Master 127.5.83.62:39395 was elected leader, sending a full tablet report...
W20260808 04:04:42.518397 15668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:04:42.518640 15668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:04:42.518743 15668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:04:42.525631 15668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:04:42.525730 15668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:04:42.525854 15668 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.3
I20260808 04:04:42.534950 15668 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:46157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/krb5kdc/oryx_127.5.83.3.keytab
--principal=oryx/127.5.83.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.3:46513
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.83.3
--webserver_port=39401
--tserver_master_addrs=127.5.83.62:39395,127.5.83.61:41591,127.5.83.60:34729
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.3
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:04:42.536116 15668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:04:42.537323 15668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:04:42.549537 15681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:42.549666 15678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:42.549942 15677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:04:42.550323 15668 server_base.cc:1034] running on GCE node
I20260808 04:04:42.550843 15668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:04:42.551627 15668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:04:42.552841 15668 hybrid_clock.cc:648] HybridClock initialized: now 1786161882552790 us; error 29 us; skew 500 ppm
I20260808 04:04:42.553143 15668 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:04:42 dist-test-slave-1l3l krb5kdc[13966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161882, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.83.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:04:42.556128 15668 init.cc:377] Logged in from keytab as oryx/127.5.83.3@KRBTEST.COM (short username oryx)
I20260808 04:04:42.557466 15668 webserver.cc:460] Webserver started at http://127.5.83.3:39401/ using document root <none> and password file <none>
I20260808 04:04:42.558082 15668 fs_manager.cc:359] Metadata directory not provided
I20260808 04:04:42.558194 15668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:04:42.563102 15668 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260808 04:04:42.565598 15688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:42.566787 15668 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:04:42.566924 15668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-2/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-2/wal
uuid: "4e9480f289294032b1d7d1018594001c"
format_stamp: "Formatted at 2026-08-08 04:04:39 on dist-test-slave-1l3l"
server_key: "90885313ee48aa73315ec34be597f735"
server_key_iv: "2246ad0aa5a18d9e45cd9c695b6e12fa"
server_key_version: "encryptionkey@0"
I20260808 04:04:42.567862 15668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:04:42.585424 15668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:04:42.589319 15668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:04:42.589533 15668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:04:42.590425 15668 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:04:42.591645 15668 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:04:42.591763 15668 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:42.591861 15668 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:04:42.591924 15668 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:42.621742 15668 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.3:46513
I20260808 04:04:42.621810 15852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.3:46513 every 8 connection(s)
I20260808 04:04:42.623124 15668 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
I20260808 04:04:42.625958 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 15668
Aug 08 04:04:42 dist-test-slave-1l3l krb5kdc[13966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161882, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.83.3@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM
Aug 08 04:04:42 dist-test-slave-1l3l krb5kdc[13966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161882, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.83.3@KRBTEST.COM for oryx/127.5.83.60@KRBTEST.COM
I20260808 04:04:42.669641 15863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:42.625986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.3:45365 (local address 127.5.83.62:39395)
0808 04:04:42.643997 (+ 18011us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:42.644004 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:42.644031 (+ 27us) server_negotiation.cc:407] Connection header received
0808 04:04:42.644105 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:42.644111 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:42.644197 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:42.644310 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:42.647349 (+ 3039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:42.648500 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:42.652671 (+ 4171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:42.653464 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:42.653593 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:42.658420 (+ 4827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:42.658427 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:42.658432 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:42.658475 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:42.662313 (+ 3838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:42.664672 (+ 2359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:42.664677 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:42.664680 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:42.664841 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:42.665413 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:42.665419 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:42.665423 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:42.665800 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:42.665937 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:42.668930 (+ 2993us) server_negotiation.cc:299] Negotiation successful
0808 04:04:42.669199 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17860,"thread_start_us":17724,"threads_started":1}
I20260808 04:04:42.673822 15867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:42.626607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:39395 (local address 127.5.83.3:45365)
0808 04:04:42.642707 (+ 16100us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:42.642728 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:42.642860 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:42.644580 (+ 1720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:42.644589 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:42.646897 (+ 2308us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:04:42.647171 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:42.647181 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:42.648686 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:42.648695 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:42.652543 (+ 3848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:42.652558 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:42.653673 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:42.653680 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:42.653861 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:42.654604 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:04:42.654637 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:04:42.658204 (+ 3567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:04:42.662714 (+ 4510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:42.662725 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:42.662744 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:42.663401 (+ 657us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:42.665009 (+ 1608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:42.665014 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:42.665018 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:42.665280 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:42.666056 (+ 776us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:04:42.666065 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:04:42.666490 (+ 425us) client_negotiation.cc:769] Sending connection context
0808 04:04:42.670382 (+ 3892us) client_negotiation.cc:240] Negotiation successful
0808 04:04:42.671019 (+ 637us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15936,"mutex_wait_us":2110,"thread_start_us":15562,"threads_started":1}
Aug 08 04:04:42 dist-test-slave-1l3l krb5kdc[13966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161882, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.83.3@KRBTEST.COM for oryx/127.5.83.61@KRBTEST.COM
I20260808 04:04:42.677752 15873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:42.632060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.3:34675 (local address 127.5.83.60:34729)
0808 04:04:42.639778 (+ 7718us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:42.639785 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:42.644532 (+ 4747us) server_negotiation.cc:407] Connection header received
0808 04:04:42.644614 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:42.644619 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:42.644704 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:42.644810 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:42.646877 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:42.648312 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:42.651666 (+ 3354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:42.652312 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:42.652449 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:42.661713 (+ 9264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:42.661720 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:42.661724 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:42.661766 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:42.668649 (+ 6883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:42.674500 (+ 5851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:42.674506 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:42.674510 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:42.674671 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:42.675232 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:42.675238 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:42.675242 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:42.675635 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:42.676961 (+ 1326us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:42.677127 (+ 166us) server_negotiation.cc:299] Negotiation successful
0808 04:04:42.677369 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7564,"thread_start_us":7436,"threads_started":1}
I20260808 04:04:42.676815 15872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:42.631999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.60:34729 (local address 127.5.83.3:34675)
0808 04:04:42.644245 (+ 12246us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:42.644265 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:42.644409 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:42.644871 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:42.644876 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:42.645659 (+ 783us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:04:42.646663 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:42.646690 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:42.648555 (+ 1865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:42.648562 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:42.651363 (+ 2801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:42.651378 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:42.652744 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:42.652750 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:42.652935 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:42.657582 (+ 4647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:04:42.657604 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:04:42.661502 (+ 3898us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:04:42.672032 (+ 10530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:42.672038 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:42.672042 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:42.672911 (+ 869us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:42.674804 (+ 1893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:42.674810 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:42.674814 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:42.675079 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:42.675814 (+ 735us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:04:42.675820 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:04:42.675976 (+ 156us) client_negotiation.cc:769] Sending connection context
0808 04:04:42.676102 (+ 126us) client_negotiation.cc:240] Negotiation successful
0808 04:04:42.676385 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12128,"mutex_wait_us":422,"thread_start_us":11989,"threads_started":1}
I20260808 04:04:42.686900 15866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:42.626607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.61:41591 (local address 127.5.83.3:38411)
0808 04:04:42.636338 (+ 9731us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:42.636400 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:42.639032 (+ 2632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:42.648069 (+ 9037us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:42.648074 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:42.649205 (+ 1131us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:04:42.649578 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:42.649591 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:42.650923 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:42.650928 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:42.657273 (+ 6345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:42.657292 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:42.672141 (+ 14849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:42.672148 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:42.672355 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:42.672955 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:04:42.672977 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:04:42.675681 (+ 2704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:04:42.683459 (+ 7778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:42.683464 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:42.683469 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:42.683963 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:42.684798 (+ 835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:42.684804 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:42.684818 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:42.685001 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:42.686172 (+ 1171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:04:42.686178 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:04:42.686317 (+ 139us) client_negotiation.cc:769] Sending connection context
0808 04:04:42.686423 (+ 106us) client_negotiation.cc:240] Negotiation successful
0808 04:04:42.686613 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9380,"mutex_wait_us":55,"thread_start_us":9152,"threads_started":1}
I20260808 04:04:42.688045 15864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:42.626073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.3:38411 (local address 127.5.83.61:41591)
0808 04:04:42.647599 (+ 21526us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:42.647605 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:42.647624 (+ 19us) server_negotiation.cc:407] Connection header received
0808 04:04:42.647679 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:42.647685 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:42.647786 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:42.647894 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:42.649881 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:42.650771 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:42.671118 (+ 20347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:42.671889 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:42.672690 (+ 801us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:42.679018 (+ 6328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:42.679026 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:42.679031 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:42.679092 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:42.682785 (+ 3693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:42.684154 (+ 1369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:42.684160 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:42.684164 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:42.684343 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:42.685160 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:42.685166 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:42.685169 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:42.685529 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:42.687298 (+ 1769us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:42.687433 (+ 135us) server_negotiation.cc:299] Negotiation successful
0808 04:04:42.687654 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21399,"thread_start_us":23231,"threads_started":1}
I20260808 04:04:42.688503 15857 heartbeater.cc:347] Connected to a master server at 127.5.83.62:39395
I20260808 04:04:42.688838 15857 heartbeater.cc:464] Registering TS with master...
I20260808 04:04:42.688984 15856 heartbeater.cc:347] Connected to a master server at 127.5.83.60:34729
I20260808 04:04:42.689121 15856 heartbeater.cc:464] Registering TS with master...
I20260808 04:04:42.689766 15856 heartbeater.cc:511] Master 127.5.83.60:34729 requested a full tablet report, sending...
I20260808 04:04:42.690550 15858 heartbeater.cc:347] Connected to a master server at 127.5.83.61:41591
I20260808 04:04:42.690685 15858 heartbeater.cc:464] Registering TS with master...
I20260808 04:04:42.691170 15857 heartbeater.cc:511] Master 127.5.83.62:39395 requested a full tablet report, sending...
I20260808 04:04:42.691608 15858 heartbeater.cc:511] Master 127.5.83.61:41591 requested a full tablet report, sending...
I20260808 04:04:42.692167 14029 ts_manager.cc:194] Re-registered known tserver with Master: 4e9480f289294032b1d7d1018594001c (127.5.83.3:46513)
I20260808 04:04:42.692797 15172 ts_manager.cc:194] Registered new tserver with Master: 4e9480f289294032b1d7d1018594001c (127.5.83.3:46513)
I20260808 04:04:42.693250 14029 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.83.3@KRBTEST.COM'} at 127.5.83.3:45365
I20260808 04:04:42.693259 14912 ts_manager.cc:194] Registered new tserver with Master: 4e9480f289294032b1d7d1018594001c (127.5.83.3:46513)
I20260808 04:04:42.703635 15873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:42.652330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59284 (local address 127.5.83.60:34729)
0808 04:04:42.679306 (+ 26976us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:42.679312 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:42.683188 (+ 3876us) server_negotiation.cc:407] Connection header received
0808 04:04:42.683246 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:42.683251 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:42.683377 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:42.683464 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:42.688286 (+ 4822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:42.689425 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:42.691333 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:42.694103 (+ 2770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:42.694208 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:42.698082 (+ 3874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:42.698088 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:42.698093 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:42.698134 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:42.701089 (+ 2955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:42.701673 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:42.701678 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:42.701682 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:42.701806 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:42.702133 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:42.702137 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:42.702140 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:42.702433 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:42.702662 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:42.703223 (+ 561us) server_negotiation.cc:299] Negotiation successful
0808 04:04:42.703373 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26872,"thread_start_us":35852,"threads_started":1}
I20260808 04:04:42.706284 15882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:42.652397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41298 (local address 127.5.83.61:41591)
0808 04:04:42.686560 (+ 34163us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:42.686566 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:42.686585 (+ 19us) server_negotiation.cc:407] Connection header received
0808 04:04:42.686636 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:42.686642 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:42.686713 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:42.686832 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:42.691280 (+ 4448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:42.692441 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:42.695609 (+ 3168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:42.696161 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:42.696271 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:42.699486 (+ 3215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:42.699491 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:42.699494 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:42.699529 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:42.702776 (+ 3247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:42.703568 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:42.703573 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:42.703577 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:42.703904 (+ 327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:42.704891 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:42.704896 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:42.704899 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:42.705257 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:42.705597 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:42.705713 (+ 116us) server_negotiation.cc:299] Negotiation successful
0808 04:04:42.705946 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33971,"thread_start_us":33829,"threads_started":1}
I20260808 04:04:42.706668 15863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:42.653369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33594 (local address 127.5.83.62:39395)
0808 04:04:42.670313 (+ 16944us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:42.670319 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:42.690072 (+ 19753us) server_negotiation.cc:407] Connection header received
0808 04:04:42.690293 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:42.690298 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:42.690381 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:42.690449 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:42.693306 (+ 2857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:42.694293 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:42.696370 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:42.696971 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:42.697064 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:42.700638 (+ 3574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:42.700644 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:42.700649 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:42.700686 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:42.703942 (+ 3256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:42.704651 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:42.704657 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:42.704660 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:42.704802 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:42.705223 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:42.705228 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:42.705231 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:42.705574 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:42.705802 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:42.706138 (+ 336us) server_negotiation.cc:299] Negotiation successful
0808 04:04:42.706348 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16823,"thread_start_us":33911,"threads_started":1}
I20260808 04:04:42.710145 14029 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33594:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:04:42.717691 14029 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:04:42.767644 15898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:42.736547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50898 (local address 127.5.83.3:46513)
0808 04:04:42.758953 (+ 22406us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:42.758963 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:42.758999 (+ 36us) server_negotiation.cc:407] Connection header received
0808 04:04:42.759107 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:42.759116 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:42.759390 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:42.759602 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:04:42.761115 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:42.762636 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:42.765140 (+ 2504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:42.766445 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:42.766577 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:42.766787 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:42.767163 (+ 376us) server_negotiation.cc:299] Negotiation successful
0808 04:04:42.767324 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22230,"thread_start_us":22065,"threads_started":1}
I20260808 04:04:42.767689 15895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:42.735510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.3:46513 (local address 127.0.0.1:50898)
0808 04:04:42.758252 (+ 22742us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:42.758279 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:42.758385 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:42.759785 (+ 1400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:42.759791 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:42.759819 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:04:42.760178 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:42.760189 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:42.762759 (+ 2570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:42.762765 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:42.764951 (+ 2186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:42.764965 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:42.766702 (+ 1737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:42.766710 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:42.766901 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:42.766972 (+ 71us) client_negotiation.cc:769] Sending connection context
0808 04:04:42.767085 (+ 113us) client_negotiation.cc:240] Negotiation successful
0808 04:04:42.767240 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22627,"mutex_wait_us":44,"thread_start_us":24768,"threads_started":1}
I20260808 04:04:42.771730 15896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:42.735124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:45257 (local address 127.0.0.1:36952)
0808 04:04:42.757415 (+ 22291us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:42.757436 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:42.757566 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:42.765095 (+ 7529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:42.765100 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:42.765126 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:04:42.765811 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:42.765825 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:42.767663 (+ 1838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:42.767668 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:42.769488 (+ 1820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:42.769502 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:42.770916 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:42.770923 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:42.771162 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:42.771186 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 04:04:42.771264 (+ 78us) client_negotiation.cc:240] Negotiation successful
0808 04:04:42.771381 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22148,"thread_start_us":27787,"threads_started":1}
I20260808 04:04:42.771780 15894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:42.735452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36952 (local address 127.5.83.1:45257)
0808 04:04:42.764210 (+ 28758us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:42.764218 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:42.764244 (+ 26us) server_negotiation.cc:407] Connection header received
0808 04:04:42.764335 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:42.764341 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:42.764873 (+ 532us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:42.765094 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:04:42.766055 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:42.767511 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:42.769656 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:42.770725 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:42.770850 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:42.771051 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:42.771320 (+ 269us) server_negotiation.cc:299] Negotiation successful
0808 04:04:42.771483 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28518,"thread_start_us":28370,"threads_started":1}
I20260808 04:04:42.771299 15753 tablet_service.cc:1467] Processing CreateTablet for tablet 6217a5b2615d4a18a1e80984530112c8 (DEFAULT_TABLE table=test-table [id=daa0d6209f724fe08ac8f359c77bc001]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:04:42.772661 15753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6217a5b2615d4a18a1e80984530112c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:04:42.775190 15337 tablet_service.cc:1467] Processing CreateTablet for tablet 6217a5b2615d4a18a1e80984530112c8 (DEFAULT_TABLE table=test-table [id=daa0d6209f724fe08ac8f359c77bc001]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:04:42.776603 15337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6217a5b2615d4a18a1e80984530112c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:04:42.778352 15893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:42.735514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.2:46681 (local address 127.0.0.1:42908)
0808 04:04:42.763240 (+ 27726us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:42.763261 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:42.763381 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:42.764237 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:42.764242 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:42.764268 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:04:42.764617 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:42.764627 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:42.766292 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:42.766297 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:42.776025 (+ 9728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:42.776038 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:42.777598 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:42.777604 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:42.777760 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:42.777792 (+ 32us) client_negotiation.cc:769] Sending connection context
0808 04:04:42.777878 (+ 86us) client_negotiation.cc:240] Negotiation successful
0808 04:04:42.777998 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27604,"mutex_wait_us":166,"thread_start_us":22000,"threads_started":1}
I20260808 04:04:42.778946 15897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:42.735935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42908 (local address 127.5.83.2:46681)
0808 04:04:42.760988 (+ 25053us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:42.760996 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:42.763381 (+ 2385us) server_negotiation.cc:407] Connection header received
0808 04:04:42.763533 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:42.763541 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:42.763820 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:42.764016 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:04:42.764798 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:42.766133 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:42.776178 (+ 10045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:42.777122 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:42.777252 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:42.777441 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:42.778614 (+ 1173us) server_negotiation.cc:299] Negotiation successful
0808 04:04:42.778743 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24895,"thread_start_us":24605,"threads_started":1}
I20260808 04:04:42.781724 15554 tablet_service.cc:1467] Processing CreateTablet for tablet 6217a5b2615d4a18a1e80984530112c8 (DEFAULT_TABLE table=test-table [id=daa0d6209f724fe08ac8f359c77bc001]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:04:42.782795 15554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6217a5b2615d4a18a1e80984530112c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:04:42.789690 15900 tablet_bootstrap.cc:492] T 6217a5b2615d4a18a1e80984530112c8 P 4e9480f289294032b1d7d1018594001c: Bootstrap starting.
I20260808 04:04:42.792147 15901 tablet_bootstrap.cc:492] T 6217a5b2615d4a18a1e80984530112c8 P 82a19f4b641448c384fdc9198ab1143e: Bootstrap starting.
I20260808 04:04:42.793882 15900 tablet_bootstrap.cc:654] T 6217a5b2615d4a18a1e80984530112c8 P 4e9480f289294032b1d7d1018594001c: Neither blocks nor log segments found. Creating new log.
I20260808 04:04:42.795164 15901 tablet_bootstrap.cc:654] T 6217a5b2615d4a18a1e80984530112c8 P 82a19f4b641448c384fdc9198ab1143e: Neither blocks nor log segments found. Creating new log.
I20260808 04:04:42.795135 15900 log.cc:824] T 6217a5b2615d4a18a1e80984530112c8 P 4e9480f289294032b1d7d1018594001c: Log is configured to *not* fsync() on all Append() calls
I20260808 04:04:42.796340 15901 log.cc:824] T 6217a5b2615d4a18a1e80984530112c8 P 82a19f4b641448c384fdc9198ab1143e: Log is configured to *not* fsync() on all Append() calls
I20260808 04:04:42.796545 15903 tablet_bootstrap.cc:492] T 6217a5b2615d4a18a1e80984530112c8 P 861039c94e114a4c9afc2caeb3282448: Bootstrap starting.
I20260808 04:04:42.798483 15900 tablet_bootstrap.cc:492] T 6217a5b2615d4a18a1e80984530112c8 P 4e9480f289294032b1d7d1018594001c: No bootstrap required, opened a new log
I20260808 04:04:42.798789 15900 ts_tablet_manager.cc:1397] T 6217a5b2615d4a18a1e80984530112c8 P 4e9480f289294032b1d7d1018594001c: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260808 04:04:42.799487 15901 tablet_bootstrap.cc:492] T 6217a5b2615d4a18a1e80984530112c8 P 82a19f4b641448c384fdc9198ab1143e: No bootstrap required, opened a new log
I20260808 04:04:42.799772 15901 ts_tablet_manager.cc:1397] T 6217a5b2615d4a18a1e80984530112c8 P 82a19f4b641448c384fdc9198ab1143e: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260808 04:04:42.800031 15903 tablet_bootstrap.cc:654] T 6217a5b2615d4a18a1e80984530112c8 P 861039c94e114a4c9afc2caeb3282448: Neither blocks nor log segments found. Creating new log.
I20260808 04:04:42.801047 15903 log.cc:824] T 6217a5b2615d4a18a1e80984530112c8 P 861039c94e114a4c9afc2caeb3282448: Log is configured to *not* fsync() on all Append() calls
I20260808 04:04:42.804404 15903 tablet_bootstrap.cc:492] T 6217a5b2615d4a18a1e80984530112c8 P 861039c94e114a4c9afc2caeb3282448: No bootstrap required, opened a new log
I20260808 04:04:42.804625 15903 ts_tablet_manager.cc:1397] T 6217a5b2615d4a18a1e80984530112c8 P 861039c94e114a4c9afc2caeb3282448: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260808 04:04:42.809367 15900 raft_consensus.cc:348] T 6217a5b2615d4a18a1e80984530112c8 P 4e9480f289294032b1d7d1018594001c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9480f289294032b1d7d1018594001c" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 46513 } } peers { permanent_uuid: "82a19f4b641448c384fdc9198ab1143e" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 45257 } } peers { permanent_uuid: "861039c94e114a4c9afc2caeb3282448" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 46681 } }
I20260808 04:04:42.809831 15900 raft_consensus.cc:374] T 6217a5b2615d4a18a1e80984530112c8 P 4e9480f289294032b1d7d1018594001c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:04:42.809943 15900 raft_consensus.cc:729] T 6217a5b2615d4a18a1e80984530112c8 P 4e9480f289294032b1d7d1018594001c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e9480f289294032b1d7d1018594001c, State: Initialized, Role: FOLLOWER
I20260808 04:04:42.810005 15901 raft_consensus.cc:348] T 6217a5b2615d4a18a1e80984530112c8 P 82a19f4b641448c384fdc9198ab1143e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9480f289294032b1d7d1018594001c" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 46513 } } peers { permanent_uuid: "82a19f4b641448c384fdc9198ab1143e" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 45257 } } peers { permanent_uuid: "861039c94e114a4c9afc2caeb3282448" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 46681 } }
I20260808 04:04:42.810480 15901 raft_consensus.cc:374] T 6217a5b2615d4a18a1e80984530112c8 P 82a19f4b641448c384fdc9198ab1143e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:04:42.810591 15901 raft_consensus.cc:729] T 6217a5b2615d4a18a1e80984530112c8 P 82a19f4b641448c384fdc9198ab1143e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82a19f4b641448c384fdc9198ab1143e, State: Initialized, Role: FOLLOWER
I20260808 04:04:42.810561 15900 consensus_queue.cc:260] T 6217a5b2615d4a18a1e80984530112c8 P 4e9480f289294032b1d7d1018594001c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9480f289294032b1d7d1018594001c" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 46513 } } peers { permanent_uuid: "82a19f4b641448c384fdc9198ab1143e" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 45257 } } peers { permanent_uuid: "861039c94e114a4c9afc2caeb3282448" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 46681 } }
I20260808 04:04:42.811252 15901 consensus_queue.cc:260] T 6217a5b2615d4a18a1e80984530112c8 P 82a19f4b641448c384fdc9198ab1143e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9480f289294032b1d7d1018594001c" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 46513 } } peers { permanent_uuid: "82a19f4b641448c384fdc9198ab1143e" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 45257 } } peers { permanent_uuid: "861039c94e114a4c9afc2caeb3282448" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 46681 } }
I20260808 04:04:42.811760 15903 raft_consensus.cc:348] T 6217a5b2615d4a18a1e80984530112c8 P 861039c94e114a4c9afc2caeb3282448 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9480f289294032b1d7d1018594001c" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 46513 } } peers { permanent_uuid: "82a19f4b641448c384fdc9198ab1143e" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 45257 } } peers { permanent_uuid: "861039c94e114a4c9afc2caeb3282448" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 46681 } }
I20260808 04:04:42.812146 15903 raft_consensus.cc:374] T 6217a5b2615d4a18a1e80984530112c8 P 861039c94e114a4c9afc2caeb3282448 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:04:42.812235 15903 raft_consensus.cc:729] T 6217a5b2615d4a18a1e80984530112c8 P 861039c94e114a4c9afc2caeb3282448 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 861039c94e114a4c9afc2caeb3282448, State: Initialized, Role: FOLLOWER
I20260808 04:04:42.812450 15900 ts_tablet_manager.cc:1428] T 6217a5b2615d4a18a1e80984530112c8 P 4e9480f289294032b1d7d1018594001c: Time spent starting tablet: real 0.014s user 0.009s sys 0.005s
I20260808 04:04:42.812733 15857 heartbeater.cc:503] Master 127.5.83.62:39395 was elected leader, sending a full tablet report...
I20260808 04:04:42.812856 15903 consensus_queue.cc:260] T 6217a5b2615d4a18a1e80984530112c8 P 861039c94e114a4c9afc2caeb3282448 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9480f289294032b1d7d1018594001c" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 46513 } } peers { permanent_uuid: "82a19f4b641448c384fdc9198ab1143e" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 45257 } } peers { permanent_uuid: "861039c94e114a4c9afc2caeb3282448" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 46681 } }
I20260808 04:04:42.820744 15901 ts_tablet_manager.cc:1428] T 6217a5b2615d4a18a1e80984530112c8 P 82a19f4b641448c384fdc9198ab1143e: Time spent starting tablet: real 0.021s user 0.010s sys 0.005s
I20260808 04:04:42.822431 15903 ts_tablet_manager.cc:1428] T 6217a5b2615d4a18a1e80984530112c8 P 861039c94e114a4c9afc2caeb3282448: Time spent starting tablet: real 0.018s user 0.013s sys 0.004s
I20260808 04:04:42.822706 15647 heartbeater.cc:503] Master 127.5.83.62:39395 was elected leader, sending a full tablet report...
W20260808 04:04:42.875965 15862 tablet.cc:2369] T 6217a5b2615d4a18a1e80984530112c8 P 4e9480f289294032b1d7d1018594001c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 04:04:42.964315 15437 tablet.cc:2369] T 6217a5b2615d4a18a1e80984530112c8 P 82a19f4b641448c384fdc9198ab1143e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 04:04:43.032042 15649 tablet.cc:2369] T 6217a5b2615d4a18a1e80984530112c8 P 861039c94e114a4c9afc2caeb3282448: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:04:43.049804 15907 raft_consensus.cc:482] T 6217a5b2615d4a18a1e80984530112c8 P 4e9480f289294032b1d7d1018594001c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:04:43.050092 15907 raft_consensus.cc:504] T 6217a5b2615d4a18a1e80984530112c8 P 4e9480f289294032b1d7d1018594001c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9480f289294032b1d7d1018594001c" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 46513 } } peers { permanent_uuid: "82a19f4b641448c384fdc9198ab1143e" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 45257 } } peers { permanent_uuid: "861039c94e114a4c9afc2caeb3282448" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 46681 } }
I20260808 04:04:43.051744 15907 leader_election.cc:288] T 6217a5b2615d4a18a1e80984530112c8 P 4e9480f289294032b1d7d1018594001c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82a19f4b641448c384fdc9198ab1143e (127.5.83.1:45257), 861039c94e114a4c9afc2caeb3282448 (127.5.83.2:46681)
I20260808 04:04:43.057571 15897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:43.052131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.3:43139 (local address 127.5.83.2:46681)
0808 04:04:43.052319 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:43.052325 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:43.052433 (+ 108us) server_negotiation.cc:407] Connection header received
0808 04:04:43.052650 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:43.052656 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:43.052723 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:43.052789 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:04:43.053236 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:43.053983 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:43.055795 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:43.056618 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:43.056754 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:43.056843 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:43.057206 (+ 363us) server_negotiation.cc:299] Negotiation successful
0808 04:04:43.057329 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 04:04:43.057576 15894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:43.052132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.3:33591 (local address 127.5.83.1:45257)
0808 04:04:43.052365 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:43.052369 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:43.052556 (+ 187us) server_negotiation.cc:407] Connection header received
0808 04:04:43.052688 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:43.052692 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:43.052764 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:43.052857 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:04:43.053278 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:43.053984 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:43.055872 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:43.056746 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:43.056879 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:43.056980 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:43.057221 (+ 241us) server_negotiation.cc:299] Negotiation successful
0808 04:04:43.057342 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260808 04:04:43.057648 15866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:43.052092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:45257 (local address 127.5.83.3:33591)
0808 04:04:43.052470 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:43.052484 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:43.052586 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:43.052886 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:43.052890 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:43.052982 (+ 92us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:04:43.053148 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:43.053155 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:43.054116 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:43.054120 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:43.055649 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:43.055661 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:43.056919 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:43.056923 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:43.057077 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:43.057100 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 04:04:43.057182 (+ 82us) client_negotiation.cc:240] Negotiation successful
0808 04:04:43.057311 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"mutex_wait_us":104,"spinlock_wait_cycles":13184}
I20260808 04:04:43.057653 15867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:43.052049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.2:46681 (local address 127.5.83.3:43139)
0808 04:04:43.052341 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:43.052361 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:43.052494 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:43.052825 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:43.052830 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:43.052855 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:04:43.053105 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:43.053115 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:43.054114 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:43.054119 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:43.055626 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:43.055641 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:43.056777 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:43.056783 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:43.057023 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:43.057085 (+ 62us) client_negotiation.cc:769] Sending connection context
0808 04:04:43.057156 (+ 71us) client_negotiation.cc:240] Negotiation successful
0808 04:04:43.057290 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":64}
I20260808 04:04:43.058745 15580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6217a5b2615d4a18a1e80984530112c8" candidate_uuid: "4e9480f289294032b1d7d1018594001c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "861039c94e114a4c9afc2caeb3282448" is_pre_election: true
I20260808 04:04:43.058894 15366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6217a5b2615d4a18a1e80984530112c8" candidate_uuid: "4e9480f289294032b1d7d1018594001c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82a19f4b641448c384fdc9198ab1143e" is_pre_election: true
I20260808 04:04:43.059338 15580 raft_consensus.cc:2454] T 6217a5b2615d4a18a1e80984530112c8 P 861039c94e114a4c9afc2caeb3282448 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e9480f289294032b1d7d1018594001c in term 0.
I20260808 04:04:43.059372 15366 raft_consensus.cc:2454] T 6217a5b2615d4a18a1e80984530112c8 P 82a19f4b641448c384fdc9198ab1143e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e9480f289294032b1d7d1018594001c in term 0.
I20260808 04:04:43.060014 15693 leader_election.cc:302] T 6217a5b2615d4a18a1e80984530112c8 P 4e9480f289294032b1d7d1018594001c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e9480f289294032b1d7d1018594001c, 861039c94e114a4c9afc2caeb3282448; no voters:
I20260808 04:04:43.060366 15907 raft_consensus.cc:2781] T 6217a5b2615d4a18a1e80984530112c8 P 4e9480f289294032b1d7d1018594001c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:04:43.060525 15907 raft_consensus.cc:482] T 6217a5b2615d4a18a1e80984530112c8 P 4e9480f289294032b1d7d1018594001c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:04:43.060626 15907 raft_consensus.cc:3037] T 6217a5b2615d4a18a1e80984530112c8 P 4e9480f289294032b1d7d1018594001c [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:04:43.062857 15907 raft_consensus.cc:504] T 6217a5b2615d4a18a1e80984530112c8 P 4e9480f289294032b1d7d1018594001c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9480f289294032b1d7d1018594001c" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 46513 } } peers { permanent_uuid: "82a19f4b641448c384fdc9198ab1143e" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 45257 } } peers { permanent_uuid: "861039c94e114a4c9afc2caeb3282448" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 46681 } }
I20260808 04:04:43.063362 15907 leader_election.cc:288] T 6217a5b2615d4a18a1e80984530112c8 P 4e9480f289294032b1d7d1018594001c [CANDIDATE]: Term 1 election: Requested vote from peers 82a19f4b641448c384fdc9198ab1143e (127.5.83.1:45257), 861039c94e114a4c9afc2caeb3282448 (127.5.83.2:46681)
I20260808 04:04:43.063784 15366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6217a5b2615d4a18a1e80984530112c8" candidate_uuid: "4e9480f289294032b1d7d1018594001c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82a19f4b641448c384fdc9198ab1143e"
I20260808 04:04:43.063840 15580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6217a5b2615d4a18a1e80984530112c8" candidate_uuid: "4e9480f289294032b1d7d1018594001c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "861039c94e114a4c9afc2caeb3282448"
I20260808 04:04:43.064007 15366 raft_consensus.cc:3037] T 6217a5b2615d4a18a1e80984530112c8 P 82a19f4b641448c384fdc9198ab1143e [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:04:43.064047 15580 raft_consensus.cc:3037] T 6217a5b2615d4a18a1e80984530112c8 P 861039c94e114a4c9afc2caeb3282448 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:04:43.067188 15580 raft_consensus.cc:2454] T 6217a5b2615d4a18a1e80984530112c8 P 861039c94e114a4c9afc2caeb3282448 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e9480f289294032b1d7d1018594001c in term 1.
I20260808 04:04:43.067269 15366 raft_consensus.cc:2454] T 6217a5b2615d4a18a1e80984530112c8 P 82a19f4b641448c384fdc9198ab1143e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e9480f289294032b1d7d1018594001c in term 1.
I20260808 04:04:43.067595 15693 leader_election.cc:302] T 6217a5b2615d4a18a1e80984530112c8 P 4e9480f289294032b1d7d1018594001c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e9480f289294032b1d7d1018594001c, 861039c94e114a4c9afc2caeb3282448; no voters:
I20260808 04:04:43.067921 15907 raft_consensus.cc:2781] T 6217a5b2615d4a18a1e80984530112c8 P 4e9480f289294032b1d7d1018594001c [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:04:43.068279 15907 raft_consensus.cc:686] T 6217a5b2615d4a18a1e80984530112c8 P 4e9480f289294032b1d7d1018594001c [term 1 LEADER]: Becoming Leader. State: Replica: 4e9480f289294032b1d7d1018594001c, State: Running, Role: LEADER
I20260808 04:04:43.068981 15907 consensus_queue.cc:237] T 6217a5b2615d4a18a1e80984530112c8 P 4e9480f289294032b1d7d1018594001c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9480f289294032b1d7d1018594001c" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 46513 } } peers { permanent_uuid: "82a19f4b641448c384fdc9198ab1143e" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 45257 } } peers { permanent_uuid: "861039c94e114a4c9afc2caeb3282448" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 46681 } }
I20260808 04:04:43.073941 14028 catalog_manager.cc:5495] T 6217a5b2615d4a18a1e80984530112c8 P 4e9480f289294032b1d7d1018594001c reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e9480f289294032b1d7d1018594001c (127.5.83.3). New cstate: current_term: 1 leader_uuid: "4e9480f289294032b1d7d1018594001c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9480f289294032b1d7d1018594001c" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 46513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82a19f4b641448c384fdc9198ab1143e" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 45257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "861039c94e114a4c9afc2caeb3282448" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 46681 } health_report { overall_health: UNKNOWN } } }
I20260808 04:04:43.115554 15898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:43.110449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50900 (local address 127.5.83.3:46513)
0808 04:04:43.110641 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:43.110647 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:43.110666 (+ 19us) server_negotiation.cc:407] Connection header received
0808 04:04:43.110847 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:43.110851 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:43.110940 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:43.111069 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:04:43.111437 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:43.112189 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:43.113509 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:43.113911 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:43.114020 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:43.114492 (+ 472us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:04:43.114966 (+ 474us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:04:43.115096 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:43.115208 (+ 112us) server_negotiation.cc:299] Negotiation successful
0808 04:04:43.115290 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 04:04:43.123436 15580 raft_consensus.cc:1261] T 6217a5b2615d4a18a1e80984530112c8 P 861039c94e114a4c9afc2caeb3282448 [term 1 FOLLOWER]: Refusing update from remote peer 4e9480f289294032b1d7d1018594001c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:04:43.123467 15366 raft_consensus.cc:1261] T 6217a5b2615d4a18a1e80984530112c8 P 82a19f4b641448c384fdc9198ab1143e [term 1 FOLLOWER]: Refusing update from remote peer 4e9480f289294032b1d7d1018594001c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:04:43.124392 15907 consensus_queue.cc:1035] T 6217a5b2615d4a18a1e80984530112c8 P 4e9480f289294032b1d7d1018594001c [LEADER]: Connected to new peer: Peer: permanent_uuid: "82a19f4b641448c384fdc9198ab1143e" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 45257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:04:43.124701 15912 consensus_queue.cc:1035] T 6217a5b2615d4a18a1e80984530112c8 P 4e9480f289294032b1d7d1018594001c [LEADER]: Connected to new peer: Peer: permanent_uuid: "861039c94e114a4c9afc2caeb3282448" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 46681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:04:43.157712 14028 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33594:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:04:43.158162 14028 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33594:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:04:43.165508 14028 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b254403e06ed4a0fbad7433dbbb9f425: Sending DeleteTablet for 3 replicas of tablet 6217a5b2615d4a18a1e80984530112c8
I20260808 04:04:43.166633 15337 tablet_service.cc:1514] Processing DeleteTablet for tablet 6217a5b2615d4a18a1e80984530112c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:04:43 UTC) from {username='oryx', principal='oryx/127.5.83.62@KRBTEST.COM'} at 127.0.0.1:36952
I20260808 04:04:43.166680 15753 tablet_service.cc:1514] Processing DeleteTablet for tablet 6217a5b2615d4a18a1e80984530112c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:04:43 UTC) from {username='oryx', principal='oryx/127.5.83.62@KRBTEST.COM'} at 127.0.0.1:50898
I20260808 04:04:43.167645 15930 tablet_replica.cc:330] T 6217a5b2615d4a18a1e80984530112c8 P 4e9480f289294032b1d7d1018594001c: stopping tablet replica
I20260808 04:04:43.167970 15554 tablet_service.cc:1514] Processing DeleteTablet for tablet 6217a5b2615d4a18a1e80984530112c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:04:43 UTC) from {username='oryx', principal='oryx/127.5.83.62@KRBTEST.COM'} at 127.0.0.1:42908
I20260808 04:04:43.168243 15930 raft_consensus.cc:2229] T 6217a5b2615d4a18a1e80984530112c8 P 4e9480f289294032b1d7d1018594001c [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:04:43.168972 15930 raft_consensus.cc:2258] T 6217a5b2615d4a18a1e80984530112c8 P 4e9480f289294032b1d7d1018594001c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:04:43.169852 15929 tablet_replica.cc:330] T 6217a5b2615d4a18a1e80984530112c8 P 82a19f4b641448c384fdc9198ab1143e: stopping tablet replica
I20260808 04:04:43.171389 15929 raft_consensus.cc:2229] T 6217a5b2615d4a18a1e80984530112c8 P 82a19f4b641448c384fdc9198ab1143e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:04:43.172039 15929 raft_consensus.cc:2258] T 6217a5b2615d4a18a1e80984530112c8 P 82a19f4b641448c384fdc9198ab1143e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:04:43.172202 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 13985
I20260808 04:04:43.173237 14082 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:04:43.174264 15929 ts_tablet_manager.cc:1905] T 6217a5b2615d4a18a1e80984530112c8 P 82a19f4b641448c384fdc9198ab1143e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:04:43.182120 15929 ts_tablet_manager.cc:1918] T 6217a5b2615d4a18a1e80984530112c8 P 82a19f4b641448c384fdc9198ab1143e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:04:43.182394 15929 log.cc:1196] T 6217a5b2615d4a18a1e80984530112c8 P 82a19f4b641448c384fdc9198ab1143e: Deleting WAL directory at /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-0/wal/wals/6217a5b2615d4a18a1e80984530112c8
I20260808 04:04:43.183029 15929 ts_tablet_manager.cc:1939] T 6217a5b2615d4a18a1e80984530112c8 P 82a19f4b641448c384fdc9198ab1143e: Deleting consensus metadata
I20260808 04:04:43.188687 15930 ts_tablet_manager.cc:1905] T 6217a5b2615d4a18a1e80984530112c8 P 4e9480f289294032b1d7d1018594001c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:04:43.191418 15882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:43.169456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41300 (local address 127.5.83.61:41591)
0808 04:04:43.169616 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:43.169639 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:43.171432 (+ 1793us) server_negotiation.cc:407] Connection header received
0808 04:04:43.172678 (+ 1246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:43.172683 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:43.172764 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:43.172841 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:43.175177 (+ 2336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:43.176312 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:43.178903 (+ 2591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:43.179857 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:43.179958 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:43.184133 (+ 4175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:43.184140 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:43.184145 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:43.184185 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:43.188008 (+ 3823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:43.188875 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:43.188881 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:43.188885 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:43.189047 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:43.189526 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:43.189532 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:43.189536 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:43.189909 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:43.190145 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:43.190772 (+ 627us) server_negotiation.cc:299] Negotiation successful
0808 04:04:43.191003 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 04:04:43.197362 15930 ts_tablet_manager.cc:1918] T 6217a5b2615d4a18a1e80984530112c8 P 4e9480f289294032b1d7d1018594001c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:04:43.197600 15930 log.cc:1196] T 6217a5b2615d4a18a1e80984530112c8 P 4e9480f289294032b1d7d1018594001c: Deleting WAL directory at /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-2/wal/wals/6217a5b2615d4a18a1e80984530112c8
I20260808 04:04:43.198436 15930 ts_tablet_manager.cc:1939] T 6217a5b2615d4a18a1e80984530112c8 P 4e9480f289294032b1d7d1018594001c: Deleting consensus metadata
I20260808 04:04:43.200743 15931 tablet_replica.cc:330] T 6217a5b2615d4a18a1e80984530112c8 P 861039c94e114a4c9afc2caeb3282448: stopping tablet replica
I20260808 04:04:43.201131 15931 raft_consensus.cc:2229] T 6217a5b2615d4a18a1e80984530112c8 P 861039c94e114a4c9afc2caeb3282448 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:04:43.201619 15931 raft_consensus.cc:2258] T 6217a5b2615d4a18a1e80984530112c8 P 861039c94e114a4c9afc2caeb3282448 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:04:43.203603 15931 ts_tablet_manager.cc:1905] T 6217a5b2615d4a18a1e80984530112c8 P 861039c94e114a4c9afc2caeb3282448: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:04:43.210109 15873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:43.171998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59286 (local address 127.5.83.60:34729)
0808 04:04:43.172172 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:43.172178 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:43.193752 (+ 21574us) server_negotiation.cc:407] Connection header received
0808 04:04:43.193903 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:43.193909 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:43.193992 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:43.194105 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:43.195763 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:43.196885 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:43.198812 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:43.199405 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:43.199534 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:43.202806 (+ 3272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:43.202813 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:43.202828 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:43.202870 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:43.206555 (+ 3685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:43.207473 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:43.207479 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:43.207482 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:43.207647 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:43.208273 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:43.208279 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:43.208282 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:43.208655 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:43.208885 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:43.209507 (+ 622us) server_negotiation.cc:299] Negotiation successful
0808 04:04:43.209732 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 04:04:43.211200 15931 ts_tablet_manager.cc:1918] T 6217a5b2615d4a18a1e80984530112c8 P 861039c94e114a4c9afc2caeb3282448: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:04:43.211373 15931 log.cc:1196] T 6217a5b2615d4a18a1e80984530112c8 P 861039c94e114a4c9afc2caeb3282448: Deleting WAL directory at /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-1/wal/wals/6217a5b2615d4a18a1e80984530112c8
I20260808 04:04:43.211948 15931 ts_tablet_manager.cc:1939] T 6217a5b2615d4a18a1e80984530112c8 P 861039c94e114a4c9afc2caeb3282448: Deleting consensus metadata
I20260808 04:04:43.325963 14011 catalog_manager.cc:4841] TS 4e9480f289294032b1d7d1018594001c (127.5.83.3:46513): tablet 6217a5b2615d4a18a1e80984530112c8 (table test-table [id=daa0d6209f724fe08ac8f359c77bc001]) successfully deleted
I20260808 04:04:43.328264 14014 catalog_manager.cc:4841] TS 861039c94e114a4c9afc2caeb3282448 (127.5.83.2:46681): tablet 6217a5b2615d4a18a1e80984530112c8 (table test-table [id=daa0d6209f724fe08ac8f359c77bc001]) successfully deleted
I20260808 04:04:43.329149 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 13985
I20260808 04:04:43.382434 15882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:43.363930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41304 (local address 127.5.83.61:41591)
0808 04:04:43.364119 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:43.364125 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:43.367027 (+ 2902us) server_negotiation.cc:407] Connection header received
0808 04:04:43.367147 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:43.367152 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:43.367236 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:43.367348 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:43.370653 (+ 3305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:43.371823 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:43.373479 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:43.374118 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:43.374243 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:43.376905 (+ 2662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:43.376912 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:43.376917 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:43.376984 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:43.379644 (+ 2660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:43.380345 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:43.380350 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:43.380352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:43.380471 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:43.380865 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:43.380869 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:43.380871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:43.381205 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:43.381440 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:43.381871 (+ 431us) server_negotiation.cc:299] Negotiation successful
0808 04:04:43.382086 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260808 04:04:43.383318 15873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:43.364451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59288 (local address 127.5.83.60:34729)
0808 04:04:43.364648 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:43.364654 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:43.366599 (+ 1945us) server_negotiation.cc:407] Connection header received
0808 04:04:43.366925 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:43.366932 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:43.367019 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:43.367095 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:43.369367 (+ 2272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:43.370479 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:43.371973 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:43.372623 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:43.372751 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:43.375869 (+ 3118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:43.375876 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:43.375880 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:43.375923 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:43.380214 (+ 4291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:43.381148 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:43.381153 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:43.381157 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:43.381301 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:43.381737 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:43.381741 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:43.381744 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:43.382128 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:43.382326 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:43.382789 (+ 463us) server_negotiation.cc:299] Negotiation successful
0808 04:04:43.383015 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 04:04:44.147850 15944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:44.146692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:39395 (local address 127.5.83.1:55651)
0808 04:04:44.147478 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:44.147625 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:39395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":629,"threads_started":1}
W20260808 04:04:44.148415 15434 heartbeater.cc:644] Failed to heartbeat to 127.5.83.62:39395 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.83.62:39395: connect: Connection refused (error 111)
I20260808 04:04:44.148628 15945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:44.147636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:39395 (local address 127.5.83.3:42359)
0808 04:04:44.148349 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:44.148476 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:39395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":523,"threads_started":1}
W20260808 04:04:44.148948 15857 heartbeater.cc:644] Failed to heartbeat to 127.5.83.62:39395 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.83.62:39395: connect: Connection refused (error 111)
I20260808 04:04:44.152726 15947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:44.151955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:39395 (local address 127.5.83.2:41793)
0808 04:04:44.152407 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:44.152547 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:39395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":237,"threads_started":1}
W20260808 04:04:44.153100 15647 heartbeater.cc:644] Failed to heartbeat to 127.5.83.62:39395 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.83.62:39395: connect: Connection refused (error 111)
I20260808 04:04:44.665050 15949 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0960abfe8684bd79c949f4811b75825 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b254403e06ed4a0fbad7433dbbb9f425)
I20260808 04:04:44.665333 15949 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0960abfe8684bd79c949f4811b75825 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b254403e06ed4a0fbad7433dbbb9f425" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 39395 } } peers { permanent_uuid: "72fecab4e79d4b7baa3fb285de73d640" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 41591 } attrs { promote: false } } peers { permanent_uuid: "f0960abfe8684bd79c949f4811b75825" member_type: VOTER last_known_addr { host: "127.5.83.60" port: 34729 } attrs { promote: false } }
I20260808 04:04:44.666095 15950 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72fecab4e79d4b7baa3fb285de73d640 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b254403e06ed4a0fbad7433dbbb9f425)
I20260808 04:04:44.666339 15950 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72fecab4e79d4b7baa3fb285de73d640 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b254403e06ed4a0fbad7433dbbb9f425" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 39395 } } peers { permanent_uuid: "72fecab4e79d4b7baa3fb285de73d640" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 41591 } attrs { promote: false } } peers { permanent_uuid: "f0960abfe8684bd79c949f4811b75825" member_type: VOTER last_known_addr { host: "127.5.83.60" port: 34729 } attrs { promote: false } }
I20260808 04:04:44.666922 15949 leader_election.cc:288] T 00000000000000000000000000000000 P f0960abfe8684bd79c949f4811b75825 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b254403e06ed4a0fbad7433dbbb9f425 (127.5.83.62:39395), 72fecab4e79d4b7baa3fb285de73d640 (127.5.83.61:41591)
I20260808 04:04:44.667737 15950 leader_election.cc:288] T 00000000000000000000000000000000 P 72fecab4e79d4b7baa3fb285de73d640 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b254403e06ed4a0fbad7433dbbb9f425 (127.5.83.62:39395), f0960abfe8684bd79c949f4811b75825 (127.5.83.60:34729)
I20260808 04:04:44.668736 15956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:44.667048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:39395 (local address 127.0.0.1:33996)
0808 04:04:44.668265 (+ 1217us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:44.668478 (+ 213us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:39395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1085,"threads_started":1}
W20260808 04:04:44.670779 15153 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.83.62:39395: connect: Connection refused (error 111)
I20260808 04:04:44.670959 15961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:44.669942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:39395 (local address 127.0.0.1:33998)
0808 04:04:44.670459 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:44.670704 (+ 245us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:39395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":263,"threads_started":1}
W20260808 04:04:44.671264 14895 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.83.62:39395: connect: Connection refused (error 111)
I20260808 04:04:44.676432 15952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:44.675912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:39395 (local address 127.0.0.1:34014)
0808 04:04:44.676094 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:44.676237 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:39395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260808 04:04:44.676622 15153 leader_election.cc:334] T 00000000000000000000000000000000 P f0960abfe8684bd79c949f4811b75825 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b254403e06ed4a0fbad7433dbbb9f425 (127.5.83.62:39395): Network error: Client connection negotiation failed: client connection to 127.5.83.62:39395: connect: Connection refused (error 111)
I20260808 04:04:44.677078 15961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:44.676678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:39395 (local address 127.0.0.1:34020)
0808 04:04:44.676846 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:44.676931 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:39395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260808 04:04:44.677234 14895 leader_election.cc:334] T 00000000000000000000000000000000 P 72fecab4e79d4b7baa3fb285de73d640 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b254403e06ed4a0fbad7433dbbb9f425 (127.5.83.62:39395): Network error: Client connection negotiation failed: client connection to 127.5.83.62:39395: connect: Connection refused (error 111)
I20260808 04:04:44.678295 15956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:44.667201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.61:41591 (local address 127.0.0.1:41306)
0808 04:04:44.668927 (+ 1726us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:44.668946 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:44.669096 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:44.670250 (+ 1154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:44.670255 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:44.670303 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:04:44.670579 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:44.670594 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:44.672510 (+ 1916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:44.672515 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:44.674565 (+ 2050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:44.674579 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:44.677546 (+ 2967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:44.677552 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:44.677728 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:44.677752 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 04:04:44.677836 (+ 84us) client_negotiation.cc:240] Negotiation successful
0808 04:04:44.677966 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1637,"thread_start_us":860,"threads_started":1}
I20260808 04:04:44.678352 15958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:44.667313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41306 (local address 127.5.83.61:41591)
0808 04:04:44.669917 (+ 2604us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:44.669923 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:44.669940 (+ 17us) server_negotiation.cc:407] Connection header received
0808 04:04:44.670000 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:44.670005 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:44.670071 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:44.670775 (+ 704us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:04:44.671051 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:44.672329 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:44.674758 (+ 2429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:44.677079 (+ 2321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:44.677504 (+ 425us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:44.677594 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:44.677889 (+ 295us) server_negotiation.cc:299] Negotiation successful
0808 04:04:44.678015 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2483,"thread_start_us":2311,"threads_started":1}
I20260808 04:04:44.678627 15962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:44.670261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59300 (local address 127.5.83.60:34729)
0808 04:04:44.670832 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:44.670849 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:44.670871 (+ 22us) server_negotiation.cc:407] Connection header received
0808 04:04:44.670929 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:44.670933 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:44.671690 (+ 757us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:44.672470 (+ 780us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:04:44.672741 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:44.673986 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:44.676514 (+ 2528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:44.677671 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:44.677795 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:44.677894 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:44.678173 (+ 279us) server_negotiation.cc:299] Negotiation successful
0808 04:04:44.678300 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":300,"threads_started":1}
I20260808 04:04:44.678675 15959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:44.667950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.60:34729 (local address 127.0.0.1:59300)
0808 04:04:44.669574 (+ 1624us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:44.669592 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:44.669719 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:44.671872 (+ 2153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:44.671877 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:44.671903 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:04:44.672325 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:44.672335 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:44.674149 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:44.674154 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:44.676262 (+ 2108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:44.676281 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:44.677870 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:44.677877 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:44.678023 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:44.678047 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 04:04:44.678134 (+ 87us) client_negotiation.cc:240] Negotiation successful
0808 04:04:44.678261 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1518,"mutex_wait_us":24,"thread_start_us":1380,"threads_started":1}
I20260808 04:04:44.679160 14925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f0960abfe8684bd79c949f4811b75825" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "72fecab4e79d4b7baa3fb285de73d640" is_pre_election: true
I20260808 04:04:44.679246 15189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "72fecab4e79d4b7baa3fb285de73d640" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f0960abfe8684bd79c949f4811b75825" is_pre_election: true
I20260808 04:04:44.679442 14925 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 72fecab4e79d4b7baa3fb285de73d640 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0960abfe8684bd79c949f4811b75825 in term 1.
I20260808 04:04:44.679527 15189 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f0960abfe8684bd79c949f4811b75825 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72fecab4e79d4b7baa3fb285de73d640 in term 1.
I20260808 04:04:44.679936 15151 leader_election.cc:302] T 00000000000000000000000000000000 P f0960abfe8684bd79c949f4811b75825 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 72fecab4e79d4b7baa3fb285de73d640, f0960abfe8684bd79c949f4811b75825; no voters: b254403e06ed4a0fbad7433dbbb9f425
I20260808 04:04:44.679963 14891 leader_election.cc:302] T 00000000000000000000000000000000 P 72fecab4e79d4b7baa3fb285de73d640 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 72fecab4e79d4b7baa3fb285de73d640, f0960abfe8684bd79c949f4811b75825; no voters: b254403e06ed4a0fbad7433dbbb9f425
I20260808 04:04:44.680332 15950 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 72fecab4e79d4b7baa3fb285de73d640 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 04:04:44.680491 15950 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72fecab4e79d4b7baa3fb285de73d640 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b254403e06ed4a0fbad7433dbbb9f425)
I20260808 04:04:44.680547 15949 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0960abfe8684bd79c949f4811b75825 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 04:04:44.680574 15950 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72fecab4e79d4b7baa3fb285de73d640 [term 1 FOLLOWER]: Advancing to term 2
I20260808 04:04:44.680631 15949 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0960abfe8684bd79c949f4811b75825 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b254403e06ed4a0fbad7433dbbb9f425)
I20260808 04:04:44.680688 15949 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0960abfe8684bd79c949f4811b75825 [term 1 FOLLOWER]: Advancing to term 2
I20260808 04:04:44.682655 15949 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0960abfe8684bd79c949f4811b75825 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b254403e06ed4a0fbad7433dbbb9f425" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 39395 } } peers { permanent_uuid: "72fecab4e79d4b7baa3fb285de73d640" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 41591 } attrs { promote: false } } peers { permanent_uuid: "f0960abfe8684bd79c949f4811b75825" member_type: VOTER last_known_addr { host: "127.5.83.60" port: 34729 } attrs { promote: false } }
I20260808 04:04:44.682677 15950 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72fecab4e79d4b7baa3fb285de73d640 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b254403e06ed4a0fbad7433dbbb9f425" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 39395 } } peers { permanent_uuid: "72fecab4e79d4b7baa3fb285de73d640" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 41591 } attrs { promote: false } } peers { permanent_uuid: "f0960abfe8684bd79c949f4811b75825" member_type: VOTER last_known_addr { host: "127.5.83.60" port: 34729 } attrs { promote: false } }
I20260808 04:04:44.683202 15949 leader_election.cc:288] T 00000000000000000000000000000000 P f0960abfe8684bd79c949f4811b75825 [CANDIDATE]: Term 2 election: Requested vote from peers b254403e06ed4a0fbad7433dbbb9f425 (127.5.83.62:39395), 72fecab4e79d4b7baa3fb285de73d640 (127.5.83.61:41591)
I20260808 04:04:44.683202 15950 leader_election.cc:288] T 00000000000000000000000000000000 P 72fecab4e79d4b7baa3fb285de73d640 [CANDIDATE]: Term 2 election: Requested vote from peers b254403e06ed4a0fbad7433dbbb9f425 (127.5.83.62:39395), f0960abfe8684bd79c949f4811b75825 (127.5.83.60:34729)
I20260808 04:04:44.683684 15956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:44.683291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:39395 (local address 127.0.0.1:34034)
0808 04:04:44.683448 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:44.683527 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:39395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260808 04:04:44.683688 15189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "72fecab4e79d4b7baa3fb285de73d640" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f0960abfe8684bd79c949f4811b75825"
I20260808 04:04:44.684074 15189 raft_consensus.cc:2379] T 00000000000000000000000000000000 P f0960abfe8684bd79c949f4811b75825 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 72fecab4e79d4b7baa3fb285de73d640 in current term 2: Already voted for candidate f0960abfe8684bd79c949f4811b75825 in this term.
I20260808 04:04:44.684255 15959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:44.683852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:39395 (local address 127.0.0.1:34032)
0808 04:04:44.684041 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:44.684122 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:39395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260808 04:04:44.684208 14925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f0960abfe8684bd79c949f4811b75825" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "72fecab4e79d4b7baa3fb285de73d640"
I20260808 04:04:44.684473 14925 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 72fecab4e79d4b7baa3fb285de73d640 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate f0960abfe8684bd79c949f4811b75825 in current term 2: Already voted for candidate 72fecab4e79d4b7baa3fb285de73d640 in this term.
I20260808 04:04:44.684671 15956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:44.684313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:39395 (local address 127.0.0.1:34046)
0808 04:04:44.684463 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:44.684544 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:39395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260808 04:04:44.684850 15153 leader_election.cc:334] T 00000000000000000000000000000000 P f0960abfe8684bd79c949f4811b75825 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer b254403e06ed4a0fbad7433dbbb9f425 (127.5.83.62:39395): Network error: Client connection negotiation failed: client connection to 127.5.83.62:39395: connect: Connection refused (error 111)
I20260808 04:04:44.685034 15151 leader_election.cc:302] T 00000000000000000000000000000000 P f0960abfe8684bd79c949f4811b75825 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0960abfe8684bd79c949f4811b75825; no voters: 72fecab4e79d4b7baa3fb285de73d640, b254403e06ed4a0fbad7433dbbb9f425
I20260808 04:04:44.685257 15949 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f0960abfe8684bd79c949f4811b75825 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 04:04:44.685597 15959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:44.685286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:39395 (local address 127.0.0.1:34054)
0808 04:04:44.685424 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:44.685497 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:39395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260808 04:04:44.685750 14895 leader_election.cc:334] T 00000000000000000000000000000000 P 72fecab4e79d4b7baa3fb285de73d640 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer b254403e06ed4a0fbad7433dbbb9f425 (127.5.83.62:39395): Network error: Client connection negotiation failed: client connection to 127.5.83.62:39395: connect: Connection refused (error 111)
I20260808 04:04:44.685887 14895 leader_election.cc:302] T 00000000000000000000000000000000 P 72fecab4e79d4b7baa3fb285de73d640 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 72fecab4e79d4b7baa3fb285de73d640; no voters: b254403e06ed4a0fbad7433dbbb9f425, f0960abfe8684bd79c949f4811b75825
I20260808 04:04:44.686134 15950 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 72fecab4e79d4b7baa3fb285de73d640 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 04:04:45.150043 15965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:45.149109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:39395 (local address 127.5.83.1:36423)
0808 04:04:45.149692 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:45.149797 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:39395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":337,"threads_started":1}
I20260808 04:04:45.150293 15966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:45.149538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:39395 (local address 127.5.83.3:60285)
0808 04:04:45.149985 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:45.150078 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:39395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":251,"threads_started":1}
I20260808 04:04:45.154444 15970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:45.153644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:39395 (local address 127.5.83.2:46361)
0808 04:04:45.154106 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:45.154203 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:39395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":268,"threads_started":1}
I20260808 04:04:46.151654 15973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:46.150895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:39395 (local address 127.5.83.1:40307)
0808 04:04:46.151372 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:46.151471 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:39395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":257,"threads_started":1}
I20260808 04:04:46.151759 15974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:46.151022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:39395 (local address 127.5.83.3:48955)
0808 04:04:46.151453 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:46.151552 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:39395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":226,"threads_started":1}
I20260808 04:04:46.156328 15975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:46.155140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:39395 (local address 127.5.83.2:53847)
0808 04:04:46.156063 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:46.156155 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:39395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":821,"thread_start_us":714,"threads_started":1}
I20260808 04:04:46.350688 15976 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0960abfe8684bd79c949f4811b75825 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:04:46.350888 15976 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0960abfe8684bd79c949f4811b75825 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b254403e06ed4a0fbad7433dbbb9f425" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 39395 } } peers { permanent_uuid: "72fecab4e79d4b7baa3fb285de73d640" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 41591 } attrs { promote: false } } peers { permanent_uuid: "f0960abfe8684bd79c949f4811b75825" member_type: VOTER last_known_addr { host: "127.5.83.60" port: 34729 } attrs { promote: false } }
I20260808 04:04:46.351575 15976 leader_election.cc:288] T 00000000000000000000000000000000 P f0960abfe8684bd79c949f4811b75825 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b254403e06ed4a0fbad7433dbbb9f425 (127.5.83.62:39395), 72fecab4e79d4b7baa3fb285de73d640 (127.5.83.61:41591)
I20260808 04:04:46.352209 14925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f0960abfe8684bd79c949f4811b75825" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "72fecab4e79d4b7baa3fb285de73d640" is_pre_election: true
I20260808 04:04:46.352473 14925 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 72fecab4e79d4b7baa3fb285de73d640 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0960abfe8684bd79c949f4811b75825 in term 2.
I20260808 04:04:46.352916 15151 leader_election.cc:302] T 00000000000000000000000000000000 P f0960abfe8684bd79c949f4811b75825 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72fecab4e79d4b7baa3fb285de73d640, f0960abfe8684bd79c949f4811b75825; no voters:
I20260808 04:04:46.353159 15976 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0960abfe8684bd79c949f4811b75825 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 04:04:46.353231 15977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:46.351794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:39395 (local address 127.0.0.1:34278)
0808 04:04:46.352906 (+ 1112us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:46.353013 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:39395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":983,"thread_start_us":853,"threads_started":1}
I20260808 04:04:46.353267 15976 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0960abfe8684bd79c949f4811b75825 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:04:46.353358 15976 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0960abfe8684bd79c949f4811b75825 [term 2 FOLLOWER]: Advancing to term 3
I20260808 04:04:46.354674 15977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:46.354286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:39395 (local address 127.0.0.1:34284)
0808 04:04:46.354480 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:46.354557 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:39395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260808 04:04:46.354825 15153 leader_election.cc:334] T 00000000000000000000000000000000 P f0960abfe8684bd79c949f4811b75825 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b254403e06ed4a0fbad7433dbbb9f425 (127.5.83.62:39395): Network error: Client connection negotiation failed: client connection to 127.5.83.62:39395: connect: Connection refused (error 111)
I20260808 04:04:46.356230 15976 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0960abfe8684bd79c949f4811b75825 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b254403e06ed4a0fbad7433dbbb9f425" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 39395 } } peers { permanent_uuid: "72fecab4e79d4b7baa3fb285de73d640" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 41591 } attrs { promote: false } } peers { permanent_uuid: "f0960abfe8684bd79c949f4811b75825" member_type: VOTER last_known_addr { host: "127.5.83.60" port: 34729 } attrs { promote: false } }
I20260808 04:04:46.356772 15976 leader_election.cc:288] T 00000000000000000000000000000000 P f0960abfe8684bd79c949f4811b75825 [CANDIDATE]: Term 3 election: Requested vote from peers b254403e06ed4a0fbad7433dbbb9f425 (127.5.83.62:39395), 72fecab4e79d4b7baa3fb285de73d640 (127.5.83.61:41591)
I20260808 04:04:46.357168 14925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f0960abfe8684bd79c949f4811b75825" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "72fecab4e79d4b7baa3fb285de73d640"
I20260808 04:04:46.357215 15977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:46.356880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:39395 (local address 127.0.0.1:34298)
0808 04:04:46.357012 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:46.357091 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:39395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260808 04:04:46.357360 14925 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72fecab4e79d4b7baa3fb285de73d640 [term 2 FOLLOWER]: Advancing to term 3
I20260808 04:04:46.358147 15977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:46.357822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:39395 (local address 127.0.0.1:34314)
0808 04:04:46.357991 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:46.358052 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:39395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260808 04:04:46.358317 15153 leader_election.cc:334] T 00000000000000000000000000000000 P f0960abfe8684bd79c949f4811b75825 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer b254403e06ed4a0fbad7433dbbb9f425 (127.5.83.62:39395): Network error: Client connection negotiation failed: client connection to 127.5.83.62:39395: connect: Connection refused (error 111)
I20260808 04:04:46.359758 14925 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 72fecab4e79d4b7baa3fb285de73d640 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0960abfe8684bd79c949f4811b75825 in term 3.
I20260808 04:04:46.360113 15151 leader_election.cc:302] T 00000000000000000000000000000000 P f0960abfe8684bd79c949f4811b75825 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 72fecab4e79d4b7baa3fb285de73d640, f0960abfe8684bd79c949f4811b75825; no voters: b254403e06ed4a0fbad7433dbbb9f425
I20260808 04:04:46.360364 15976 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0960abfe8684bd79c949f4811b75825 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 04:04:46.360740 15976 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0960abfe8684bd79c949f4811b75825 [term 3 LEADER]: Becoming Leader. State: Replica: f0960abfe8684bd79c949f4811b75825, State: Running, Role: LEADER
I20260808 04:04:46.361088 15976 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0960abfe8684bd79c949f4811b75825 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b254403e06ed4a0fbad7433dbbb9f425" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 39395 } } peers { permanent_uuid: "72fecab4e79d4b7baa3fb285de73d640" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 41591 } attrs { promote: false } } peers { permanent_uuid: "f0960abfe8684bd79c949f4811b75825" member_type: VOTER last_known_addr { host: "127.5.83.60" port: 34729 } attrs { promote: false } }
I20260808 04:04:46.362879 15983 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0960abfe8684bd79c949f4811b75825 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0960abfe8684bd79c949f4811b75825. Latest consensus state: current_term: 3 leader_uuid: "f0960abfe8684bd79c949f4811b75825" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b254403e06ed4a0fbad7433dbbb9f425" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 39395 } } peers { permanent_uuid: "72fecab4e79d4b7baa3fb285de73d640" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 41591 } attrs { promote: false } } peers { permanent_uuid: "f0960abfe8684bd79c949f4811b75825" member_type: VOTER last_known_addr { host: "127.5.83.60" port: 34729 } attrs { promote: false } } }
I20260808 04:04:46.363149 15983 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0960abfe8684bd79c949f4811b75825 [sys.catalog]: This master's current role is: LEADER
I20260808 04:04:46.363776 15986 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:04:46.367533 15986 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:04:46.368350 15986 catalog_manager.cc:1259] Loaded cluster ID: a15e7f4b3b484d4dbf31a2eea6d58df7
I20260808 04:04:46.368428 15986 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:04:46.369554 15986 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:04:46.370316 15986 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f0960abfe8684bd79c949f4811b75825: Loaded TSK: 0
I20260808 04:04:46.371299 15986 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:04:46.390273 15169 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59288:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:04:46.396886 15169 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:04:46.400794 15977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:46.400410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:39395 (local address 127.0.0.1:34342)
0808 04:04:46.400591 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:46.400660 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:39395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260808 04:04:46.400846 14925 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 72fecab4e79d4b7baa3fb285de73d640 [term 3 FOLLOWER]: Refusing update from remote peer f0960abfe8684bd79c949f4811b75825: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260808 04:04:46.401535 15983 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f0960abfe8684bd79c949f4811b75825 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72fecab4e79d4b7baa3fb285de73d640" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 41591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260808 04:04:46.401789 15977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:46.401372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:39395 (local address 127.0.0.1:34348)
0808 04:04:46.401565 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:46.401648 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:39395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260808 04:04:46.401983 15153 consensus_peers.cc:487] T 00000000000000000000000000000000 P f0960abfe8684bd79c949f4811b75825 -> Peer b254403e06ed4a0fbad7433dbbb9f425 (127.5.83.62:39395): Couldn't send request to peer b254403e06ed4a0fbad7433dbbb9f425. Status: Network error: Client connection negotiation failed: client connection to 127.5.83.62:39395: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260808 04:04:46.403865 15992 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72fecab4e79d4b7baa3fb285de73d640 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0960abfe8684bd79c949f4811b75825. Latest consensus state: current_term: 3 leader_uuid: "f0960abfe8684bd79c949f4811b75825" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b254403e06ed4a0fbad7433dbbb9f425" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 39395 } } peers { permanent_uuid: "72fecab4e79d4b7baa3fb285de73d640" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 41591 } attrs { promote: false } } peers { permanent_uuid: "f0960abfe8684bd79c949f4811b75825" member_type: VOTER last_known_addr { host: "127.5.83.60" port: 34729 } attrs { promote: false } } }
I20260808 04:04:46.404102 15992 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72fecab4e79d4b7baa3fb285de73d640 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:04:46.405543 15976 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0960abfe8684bd79c949f4811b75825 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "f0960abfe8684bd79c949f4811b75825" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b254403e06ed4a0fbad7433dbbb9f425" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 39395 } } peers { permanent_uuid: "72fecab4e79d4b7baa3fb285de73d640" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 41591 } attrs { promote: false } } peers { permanent_uuid: "f0960abfe8684bd79c949f4811b75825" member_type: VOTER last_known_addr { host: "127.5.83.60" port: 34729 } attrs { promote: false } } }
I20260808 04:04:46.405820 15976 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0960abfe8684bd79c949f4811b75825 [sys.catalog]: This master's current role is: LEADER
I20260808 04:04:46.406023 15992 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72fecab4e79d4b7baa3fb285de73d640 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "f0960abfe8684bd79c949f4811b75825" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b254403e06ed4a0fbad7433dbbb9f425" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 39395 } } peers { permanent_uuid: "72fecab4e79d4b7baa3fb285de73d640" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 41591 } attrs { promote: false } } peers { permanent_uuid: "f0960abfe8684bd79c949f4811b75825" member_type: VOTER last_known_addr { host: "127.5.83.60" port: 34729 } attrs { promote: false } } }
I20260808 04:04:46.406275 15992 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72fecab4e79d4b7baa3fb285de73d640 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:04:46.406490 15993 mvcc.cc:205] Tried to move back new op lower bound from 7316119086691528704 to 7316119086538182656. Current Snapshot: MvccSnapshot[applied={T|T < 7316119086691528704}]
I20260808 04:04:46.421654 15977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:46.413388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:45257 (local address 127.0.0.1:36962)
0808 04:04:46.413633 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:46.413650 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:46.413766 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:46.414419 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:46.414424 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:46.414451 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:04:46.415201 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:46.415249 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:46.416724 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:46.416730 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:46.418871 (+ 2141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:46.418889 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:46.420816 (+ 1927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:46.420823 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:46.421035 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:46.421072 (+ 37us) client_negotiation.cc:769] Sending connection context
0808 04:04:46.421202 (+ 130us) client_negotiation.cc:240] Negotiation successful
0808 04:04:46.421350 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151}
I20260808 04:04:46.422768 15998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:46.413532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36962 (local address 127.5.83.1:45257)
0808 04:04:46.414017 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:46.414023 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:46.414043 (+ 20us) server_negotiation.cc:407] Connection header received
0808 04:04:46.414182 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:46.414187 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:46.414264 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:46.414726 (+ 462us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:04:46.415408 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:46.416536 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:46.419112 (+ 2576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:46.420528 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:46.420655 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:46.422223 (+ 1568us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:46.422276 (+ 53us) server_negotiation.cc:299] Negotiation successful
0808 04:04:46.422426 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":859,"threads_started":1}
I20260808 04:04:46.423784 15337 tablet_service.cc:1467] Processing CreateTablet for tablet fa1b13b41e41437783099c2477819ef6 (DEFAULT_TABLE table=test-table [id=0184e889267a4ae2aa275f982453efa2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:04:46.424301 15337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa1b13b41e41437783099c2477819ef6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:04:46.431252 16005 tablet_bootstrap.cc:492] T fa1b13b41e41437783099c2477819ef6 P 82a19f4b641448c384fdc9198ab1143e: Bootstrap starting.
I20260808 04:04:46.433166 16005 tablet_bootstrap.cc:654] T fa1b13b41e41437783099c2477819ef6 P 82a19f4b641448c384fdc9198ab1143e: Neither blocks nor log segments found. Creating new log.
I20260808 04:04:46.433247 16002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:46.413866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.2:46681 (local address 127.0.0.1:42910)
0808 04:04:46.422980 (+ 9114us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:46.423006 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:46.423120 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:46.427502 (+ 4382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:46.427508 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:46.427533 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:04:46.427798 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:46.427808 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:46.429403 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:46.429408 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:46.430645 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:46.430657 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:46.432450 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:46.432456 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:46.432652 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:46.432689 (+ 37us) client_negotiation.cc:769] Sending connection context
0808 04:04:46.432781 (+ 92us) client_negotiation.cc:240] Negotiation successful
0808 04:04:46.432911 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1348,"thread_start_us":12269,"threads_started":1}
I20260808 04:04:46.433501 16001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:46.414246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42910 (local address 127.5.83.2:46681)
0808 04:04:46.427187 (+ 12941us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:46.427193 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:46.427213 (+ 20us) server_negotiation.cc:407] Connection header received
0808 04:04:46.427285 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:46.427291 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:46.427360 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:46.427471 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:04:46.427971 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:46.429270 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:46.430797 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:46.432235 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:46.432369 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:46.432469 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:46.432824 (+ 355us) server_negotiation.cc:299] Negotiation successful
0808 04:04:46.432963 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12804,"thread_start_us":12680,"threads_started":1}
I20260808 04:04:46.434300 15554 tablet_service.cc:1467] Processing CreateTablet for tablet fa1b13b41e41437783099c2477819ef6 (DEFAULT_TABLE table=test-table [id=0184e889267a4ae2aa275f982453efa2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:04:46.434746 15554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa1b13b41e41437783099c2477819ef6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:04:46.435528 16005 tablet_bootstrap.cc:492] T fa1b13b41e41437783099c2477819ef6 P 82a19f4b641448c384fdc9198ab1143e: No bootstrap required, opened a new log
I20260808 04:04:46.435668 16005 ts_tablet_manager.cc:1397] T fa1b13b41e41437783099c2477819ef6 P 82a19f4b641448c384fdc9198ab1143e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260808 04:04:46.436455 16005 raft_consensus.cc:348] T fa1b13b41e41437783099c2477819ef6 P 82a19f4b641448c384fdc9198ab1143e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82a19f4b641448c384fdc9198ab1143e" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 45257 } } peers { permanent_uuid: "861039c94e114a4c9afc2caeb3282448" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 46681 } } peers { permanent_uuid: "4e9480f289294032b1d7d1018594001c" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 46513 } }
I20260808 04:04:46.436725 16005 raft_consensus.cc:374] T fa1b13b41e41437783099c2477819ef6 P 82a19f4b641448c384fdc9198ab1143e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:04:46.436847 16005 raft_consensus.cc:729] T fa1b13b41e41437783099c2477819ef6 P 82a19f4b641448c384fdc9198ab1143e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82a19f4b641448c384fdc9198ab1143e, State: Initialized, Role: FOLLOWER
I20260808 04:04:46.437117 16005 consensus_queue.cc:260] T fa1b13b41e41437783099c2477819ef6 P 82a19f4b641448c384fdc9198ab1143e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82a19f4b641448c384fdc9198ab1143e" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 45257 } } peers { permanent_uuid: "861039c94e114a4c9afc2caeb3282448" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 46681 } } peers { permanent_uuid: "4e9480f289294032b1d7d1018594001c" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 46513 } }
I20260808 04:04:46.437805 15977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:46.414532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.3:46513 (local address 127.0.0.1:50904)
0808 04:04:46.421877 (+ 7345us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:46.421895 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:46.422006 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:46.430797 (+ 8791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:46.430802 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:46.430829 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:04:46.430977 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:46.430988 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:46.432257 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:46.432262 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:46.435825 (+ 3563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:46.435839 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:46.437044 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:46.437049 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:46.437239 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:46.437272 (+ 33us) client_negotiation.cc:769] Sending connection context
0808 04:04:46.437416 (+ 144us) client_negotiation.cc:240] Negotiation successful
0808 04:04:46.437543 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7249,"thread_start_us":887,"threads_started":1}
I20260808 04:04:46.438690 16004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:46.417336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50904 (local address 127.5.83.3:46513)
0808 04:04:46.430365 (+ 13029us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:46.430371 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:46.430390 (+ 19us) server_negotiation.cc:407] Connection header received
0808 04:04:46.430472 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:46.430478 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:46.430554 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:46.431116 (+ 562us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:04:46.431333 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:46.432116 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:46.435987 (+ 3871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:46.436872 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:46.436986 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:46.437156 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:46.438246 (+ 1090us) server_negotiation.cc:299] Negotiation successful
0808 04:04:46.438395 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12896,"thread_start_us":13295,"threads_started":1}
I20260808 04:04:46.438813 16005 ts_tablet_manager.cc:1428] T fa1b13b41e41437783099c2477819ef6 P 82a19f4b641448c384fdc9198ab1143e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 04:04:46.439555 15753 tablet_service.cc:1467] Processing CreateTablet for tablet fa1b13b41e41437783099c2477819ef6 (DEFAULT_TABLE table=test-table [id=0184e889267a4ae2aa275f982453efa2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:04:46.439967 15973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:46.439382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:39395 (local address 127.5.83.1:33869)
0808 04:04:46.439719 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:46.439809 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:39395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206}
I20260808 04:04:46.440055 15753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa1b13b41e41437783099c2477819ef6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:04:46.444093 16009 tablet_bootstrap.cc:492] T fa1b13b41e41437783099c2477819ef6 P 861039c94e114a4c9afc2caeb3282448: Bootstrap starting.
I20260808 04:04:46.444278 15432 heartbeater.cc:503] Master 127.5.83.60:34729 was elected leader, sending a full tablet report...
I20260808 04:04:46.445941 15973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:46.445494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:39395 (local address 127.5.83.1:44357)
0808 04:04:46.445694 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:46.445792 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:39395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260808 04:04:46.447479 16009 tablet_bootstrap.cc:654] T fa1b13b41e41437783099c2477819ef6 P 861039c94e114a4c9afc2caeb3282448: Neither blocks nor log segments found. Creating new log.
I20260808 04:04:46.449499 16009 tablet_bootstrap.cc:492] T fa1b13b41e41437783099c2477819ef6 P 861039c94e114a4c9afc2caeb3282448: No bootstrap required, opened a new log
I20260808 04:04:46.449628 16009 ts_tablet_manager.cc:1397] T fa1b13b41e41437783099c2477819ef6 P 861039c94e114a4c9afc2caeb3282448: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20260808 04:04:46.450222 16009 raft_consensus.cc:348] T fa1b13b41e41437783099c2477819ef6 P 861039c94e114a4c9afc2caeb3282448 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82a19f4b641448c384fdc9198ab1143e" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 45257 } } peers { permanent_uuid: "861039c94e114a4c9afc2caeb3282448" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 46681 } } peers { permanent_uuid: "4e9480f289294032b1d7d1018594001c" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 46513 } }
I20260808 04:04:46.450428 16009 raft_consensus.cc:374] T fa1b13b41e41437783099c2477819ef6 P 861039c94e114a4c9afc2caeb3282448 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:04:46.450528 16009 raft_consensus.cc:729] T fa1b13b41e41437783099c2477819ef6 P 861039c94e114a4c9afc2caeb3282448 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 861039c94e114a4c9afc2caeb3282448, State: Initialized, Role: FOLLOWER
I20260808 04:04:46.450543 16010 tablet_bootstrap.cc:492] T fa1b13b41e41437783099c2477819ef6 P 4e9480f289294032b1d7d1018594001c: Bootstrap starting.
I20260808 04:04:46.450796 16009 consensus_queue.cc:260] T fa1b13b41e41437783099c2477819ef6 P 861039c94e114a4c9afc2caeb3282448 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82a19f4b641448c384fdc9198ab1143e" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 45257 } } peers { permanent_uuid: "861039c94e114a4c9afc2caeb3282448" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 46681 } } peers { permanent_uuid: "4e9480f289294032b1d7d1018594001c" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 46513 } }
I20260808 04:04:46.451579 16009 ts_tablet_manager.cc:1428] T fa1b13b41e41437783099c2477819ef6 P 861039c94e114a4c9afc2caeb3282448: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260808 04:04:46.452327 16010 tablet_bootstrap.cc:654] T fa1b13b41e41437783099c2477819ef6 P 4e9480f289294032b1d7d1018594001c: Neither blocks nor log segments found. Creating new log.
I20260808 04:04:46.452679 15975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:46.452251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:39395 (local address 127.5.83.2:42639)
0808 04:04:46.452445 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:46.452538 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:39395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260808 04:04:46.453358 15646 heartbeater.cc:503] Master 127.5.83.60:34729 was elected leader, sending a full tablet report...
I20260808 04:04:46.454483 15975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:46.454073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:39395 (local address 127.5.83.2:34891)
0808 04:04:46.454267 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:46.454343 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:39395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260808 04:04:46.455356 16010 tablet_bootstrap.cc:492] T fa1b13b41e41437783099c2477819ef6 P 4e9480f289294032b1d7d1018594001c: No bootstrap required, opened a new log
I20260808 04:04:46.455551 16010 ts_tablet_manager.cc:1397] T fa1b13b41e41437783099c2477819ef6 P 4e9480f289294032b1d7d1018594001c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260808 04:04:46.456409 16010 raft_consensus.cc:348] T fa1b13b41e41437783099c2477819ef6 P 4e9480f289294032b1d7d1018594001c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82a19f4b641448c384fdc9198ab1143e" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 45257 } } peers { permanent_uuid: "861039c94e114a4c9afc2caeb3282448" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 46681 } } peers { permanent_uuid: "4e9480f289294032b1d7d1018594001c" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 46513 } }
I20260808 04:04:46.456653 16010 raft_consensus.cc:374] T fa1b13b41e41437783099c2477819ef6 P 4e9480f289294032b1d7d1018594001c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:04:46.456759 16010 raft_consensus.cc:729] T fa1b13b41e41437783099c2477819ef6 P 4e9480f289294032b1d7d1018594001c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e9480f289294032b1d7d1018594001c, State: Initialized, Role: FOLLOWER
I20260808 04:04:46.457036 16010 consensus_queue.cc:260] T fa1b13b41e41437783099c2477819ef6 P 4e9480f289294032b1d7d1018594001c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82a19f4b641448c384fdc9198ab1143e" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 45257 } } peers { permanent_uuid: "861039c94e114a4c9afc2caeb3282448" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 46681 } } peers { permanent_uuid: "4e9480f289294032b1d7d1018594001c" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 46513 } }
I20260808 04:04:46.457861 16010 ts_tablet_manager.cc:1428] T fa1b13b41e41437783099c2477819ef6 P 4e9480f289294032b1d7d1018594001c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 04:04:46.458856 15974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:46.458462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:39395 (local address 127.5.83.3:45647)
0808 04:04:46.458631 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:46.458719 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:39395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260808 04:04:46.460336 15974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:46.459771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:39395 (local address 127.5.83.3:45377)
0808 04:04:46.460117 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:46.460196 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:39395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245}
I20260808 04:04:46.460726 15856 heartbeater.cc:503] Master 127.5.83.60:34729 was elected leader, sending a full tablet report...
I20260808 04:04:46.681869 16007 raft_consensus.cc:482] T fa1b13b41e41437783099c2477819ef6 P 82a19f4b641448c384fdc9198ab1143e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:04:46.682138 16007 raft_consensus.cc:504] T fa1b13b41e41437783099c2477819ef6 P 82a19f4b641448c384fdc9198ab1143e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82a19f4b641448c384fdc9198ab1143e" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 45257 } } peers { permanent_uuid: "861039c94e114a4c9afc2caeb3282448" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 46681 } } peers { permanent_uuid: "4e9480f289294032b1d7d1018594001c" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 46513 } }
I20260808 04:04:46.683681 16007 leader_election.cc:288] T fa1b13b41e41437783099c2477819ef6 P 82a19f4b641448c384fdc9198ab1143e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 861039c94e114a4c9afc2caeb3282448 (127.5.83.2:46681), 4e9480f289294032b1d7d1018594001c (127.5.83.3:46513)
I20260808 04:04:46.690506 16004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:46.684089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.1:32887 (local address 127.5.83.3:46513)
0808 04:04:46.684433 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:46.684441 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:46.684931 (+ 490us) server_negotiation.cc:407] Connection header received
0808 04:04:46.685149 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:46.685153 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:46.685225 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:46.685326 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:04:46.685751 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:46.686639 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:46.688865 (+ 2226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:46.689627 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:46.689743 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:46.689843 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:46.690136 (+ 293us) server_negotiation.cc:299] Negotiation successful
0808 04:04:46.690262 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260808 04:04:46.690649 16001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:46.684108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.1:40069 (local address 127.5.83.2:46681)
0808 04:04:46.684335 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:46.684341 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:46.684364 (+ 23us) server_negotiation.cc:407] Connection header received
0808 04:04:46.684578 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:46.684584 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:46.684662 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:46.684765 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:04:46.685363 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:46.686362 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:46.688848 (+ 2486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:46.689722 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:46.689850 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:46.689952 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:46.690271 (+ 319us) server_negotiation.cc:299] Negotiation successful
0808 04:04:46.690405 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 04:04:46.690646 16017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:46.683949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.3:46513 (local address 127.5.83.1:32887)
0808 04:04:46.684828 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:46.684844 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:46.684963 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:46.685345 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:46.685349 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:46.685369 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:04:46.685609 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:46.685616 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:46.686759 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:46.686764 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:46.688748 (+ 1984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:46.688760 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:46.689810 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:46.689816 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:46.689989 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:46.690014 (+ 25us) client_negotiation.cc:769] Sending connection context
0808 04:04:46.690110 (+ 96us) client_negotiation.cc:240] Negotiation successful
0808 04:04:46.690236 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":757,"mutex_wait_us":143,"thread_start_us":394,"threads_started":1}
I20260808 04:04:46.690671 15973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:46.683950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.2:46681 (local address 127.5.83.1:40069)
0808 04:04:46.684162 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:46.684190 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:46.684388 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:46.684828 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:46.684833 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:46.684866 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:04:46.685155 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:46.685165 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:46.686492 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:46.686495 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:46.688651 (+ 2156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:46.688668 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:46.689974 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:46.689981 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:46.690118 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:46.690143 (+ 25us) client_negotiation.cc:769] Sending connection context
0808 04:04:46.690223 (+ 80us) client_negotiation.cc:240] Negotiation successful
0808 04:04:46.690360 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260808 04:04:46.691339 15580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1b13b41e41437783099c2477819ef6" candidate_uuid: "82a19f4b641448c384fdc9198ab1143e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "861039c94e114a4c9afc2caeb3282448" is_pre_election: true
I20260808 04:04:46.691560 15580 raft_consensus.cc:2454] T fa1b13b41e41437783099c2477819ef6 P 861039c94e114a4c9afc2caeb3282448 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82a19f4b641448c384fdc9198ab1143e in term 0.
I20260808 04:04:46.691627 15783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1b13b41e41437783099c2477819ef6" candidate_uuid: "82a19f4b641448c384fdc9198ab1143e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e9480f289294032b1d7d1018594001c" is_pre_election: true
I20260808 04:04:46.692030 15783 raft_consensus.cc:2454] T fa1b13b41e41437783099c2477819ef6 P 4e9480f289294032b1d7d1018594001c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82a19f4b641448c384fdc9198ab1143e in term 0.
I20260808 04:04:46.692066 15275 leader_election.cc:302] T fa1b13b41e41437783099c2477819ef6 P 82a19f4b641448c384fdc9198ab1143e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82a19f4b641448c384fdc9198ab1143e, 861039c94e114a4c9afc2caeb3282448; no voters:
I20260808 04:04:46.692440 16007 raft_consensus.cc:2781] T fa1b13b41e41437783099c2477819ef6 P 82a19f4b641448c384fdc9198ab1143e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:04:46.692559 16007 raft_consensus.cc:482] T fa1b13b41e41437783099c2477819ef6 P 82a19f4b641448c384fdc9198ab1143e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:04:46.692651 16007 raft_consensus.cc:3037] T fa1b13b41e41437783099c2477819ef6 P 82a19f4b641448c384fdc9198ab1143e [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:04:46.694741 16007 raft_consensus.cc:504] T fa1b13b41e41437783099c2477819ef6 P 82a19f4b641448c384fdc9198ab1143e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82a19f4b641448c384fdc9198ab1143e" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 45257 } } peers { permanent_uuid: "861039c94e114a4c9afc2caeb3282448" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 46681 } } peers { permanent_uuid: "4e9480f289294032b1d7d1018594001c" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 46513 } }
I20260808 04:04:46.695269 16007 leader_election.cc:288] T fa1b13b41e41437783099c2477819ef6 P 82a19f4b641448c384fdc9198ab1143e [CANDIDATE]: Term 1 election: Requested vote from peers 861039c94e114a4c9afc2caeb3282448 (127.5.83.2:46681), 4e9480f289294032b1d7d1018594001c (127.5.83.3:46513)
I20260808 04:04:46.695804 15783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1b13b41e41437783099c2477819ef6" candidate_uuid: "82a19f4b641448c384fdc9198ab1143e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e9480f289294032b1d7d1018594001c"
I20260808 04:04:46.695811 15580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1b13b41e41437783099c2477819ef6" candidate_uuid: "82a19f4b641448c384fdc9198ab1143e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "861039c94e114a4c9afc2caeb3282448"
I20260808 04:04:46.696004 15580 raft_consensus.cc:3037] T fa1b13b41e41437783099c2477819ef6 P 861039c94e114a4c9afc2caeb3282448 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:04:46.696004 15783 raft_consensus.cc:3037] T fa1b13b41e41437783099c2477819ef6 P 4e9480f289294032b1d7d1018594001c [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:04:46.699066 15580 raft_consensus.cc:2454] T fa1b13b41e41437783099c2477819ef6 P 861039c94e114a4c9afc2caeb3282448 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82a19f4b641448c384fdc9198ab1143e in term 1.
I20260808 04:04:46.699142 15783 raft_consensus.cc:2454] T fa1b13b41e41437783099c2477819ef6 P 4e9480f289294032b1d7d1018594001c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82a19f4b641448c384fdc9198ab1143e in term 1.
I20260808 04:04:46.699430 15275 leader_election.cc:302] T fa1b13b41e41437783099c2477819ef6 P 82a19f4b641448c384fdc9198ab1143e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82a19f4b641448c384fdc9198ab1143e, 861039c94e114a4c9afc2caeb3282448; no voters:
I20260808 04:04:46.699678 16007 raft_consensus.cc:2781] T fa1b13b41e41437783099c2477819ef6 P 82a19f4b641448c384fdc9198ab1143e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:04:46.700125 16007 raft_consensus.cc:686] T fa1b13b41e41437783099c2477819ef6 P 82a19f4b641448c384fdc9198ab1143e [term 1 LEADER]: Becoming Leader. State: Replica: 82a19f4b641448c384fdc9198ab1143e, State: Running, Role: LEADER
I20260808 04:04:46.700510 16007 consensus_queue.cc:237] T fa1b13b41e41437783099c2477819ef6 P 82a19f4b641448c384fdc9198ab1143e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82a19f4b641448c384fdc9198ab1143e" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 45257 } } peers { permanent_uuid: "861039c94e114a4c9afc2caeb3282448" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 46681 } } peers { permanent_uuid: "4e9480f289294032b1d7d1018594001c" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 46513 } }
I20260808 04:04:46.701059 15973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:46.700636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:39395 (local address 127.5.83.1:38669)
0808 04:04:46.700823 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:46.700921 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:39395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260808 04:04:46.703089 15973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:46.702657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:39395 (local address 127.5.83.1:39251)
0808 04:04:46.702857 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:46.702953 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:39395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260808 04:04:46.704160 15169 catalog_manager.cc:5495] T fa1b13b41e41437783099c2477819ef6 P 82a19f4b641448c384fdc9198ab1143e reported cstate change: term changed from 0 to 1, leader changed from <none> to 82a19f4b641448c384fdc9198ab1143e (127.5.83.1). New cstate: current_term: 1 leader_uuid: "82a19f4b641448c384fdc9198ab1143e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82a19f4b641448c384fdc9198ab1143e" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 45257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "861039c94e114a4c9afc2caeb3282448" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 46681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e9480f289294032b1d7d1018594001c" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 46513 } health_report { overall_health: UNKNOWN } } }
I20260808 04:04:46.778321 15977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:46.777860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:39395 (local address 127.0.0.1:34362)
0808 04:04:46.778027 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:46.778157 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:39395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260808 04:04:46.779291 15977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:46.778957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:39395 (local address 127.0.0.1:34366)
0808 04:04:46.779111 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:46.779180 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:39395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260808 04:04:46.789208 15998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:46.784252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36976 (local address 127.5.83.1:45257)
0808 04:04:46.784461 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:46.784467 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:46.784486 (+ 19us) server_negotiation.cc:407] Connection header received
0808 04:04:46.784686 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:46.784690 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:46.784779 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:46.784922 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:04:46.785267 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:46.785969 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:46.787224 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:46.787630 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:46.787763 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:46.788170 (+ 407us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:04:46.788667 (+ 497us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:04:46.788793 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:46.788894 (+ 101us) server_negotiation.cc:299] Negotiation successful
0808 04:04:46.788973 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 04:04:46.793223 15580 raft_consensus.cc:1261] T fa1b13b41e41437783099c2477819ef6 P 861039c94e114a4c9afc2caeb3282448 [term 1 FOLLOWER]: Refusing update from remote peer 82a19f4b641448c384fdc9198ab1143e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:04:46.793582 15783 raft_consensus.cc:1261] T fa1b13b41e41437783099c2477819ef6 P 4e9480f289294032b1d7d1018594001c [term 1 FOLLOWER]: Refusing update from remote peer 82a19f4b641448c384fdc9198ab1143e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:04:46.793942 16018 consensus_queue.cc:1035] T fa1b13b41e41437783099c2477819ef6 P 82a19f4b641448c384fdc9198ab1143e [LEADER]: Connected to new peer: Peer: permanent_uuid: "861039c94e114a4c9afc2caeb3282448" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 46681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:04:46.794350 16007 consensus_queue.cc:1035] T fa1b13b41e41437783099c2477819ef6 P 82a19f4b641448c384fdc9198ab1143e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e9480f289294032b1d7d1018594001c" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 46513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:04:46.796631 15975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:46.796111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:39395 (local address 127.5.83.2:38367)
0808 04:04:46.796347 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:46.796451 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:39395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
I20260808 04:04:46.797142 15974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:46.796597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:39395 (local address 127.5.83.3:34535)
0808 04:04:46.796852 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:46.796950 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:39395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
I20260808 04:04:46.801760 16021 mvcc.cc:205] Tried to move back new op lower bound from 7316119088297713664 to 7316119087929184256. Current Snapshot: MvccSnapshot[applied={T|T < 7316119088297713664}]
I20260808 04:04:46.804057 15973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:46.803425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:39395 (local address 127.5.83.1:39489)
0808 04:04:46.803784 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:46.803894 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:39395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232}
I20260808 04:04:46.804272 16023 mvcc.cc:205] Tried to move back new op lower bound from 7316119088297713664 to 7316119087929184256. Current Snapshot: MvccSnapshot[applied={T|T < 7316119088297713664}]
I20260808 04:04:46.804461 15974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:46.803967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:39395 (local address 127.5.83.3:42811)
0808 04:04:46.804206 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:46.804293 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:39395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260808 04:04:46.805634 15975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:46.805040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:39395 (local address 127.5.83.2:44523)
0808 04:04:46.805355 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:46.805452 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:39395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198}
I20260808 04:04:46.808686 16026 mvcc.cc:205] Tried to move back new op lower bound from 7316119088297713664 to 7316119087929184256. Current Snapshot: MvccSnapshot[applied={T|T < 7316119088297713664}]
I20260808 04:04:46.816817 15169 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59288:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:04:46.817041 15169 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59288:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:04:46.822260 15169 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f0960abfe8684bd79c949f4811b75825: Sending DeleteTablet for 3 replicas of tablet fa1b13b41e41437783099c2477819ef6
I20260808 04:04:46.823436 15337 tablet_service.cc:1514] Processing DeleteTablet for tablet fa1b13b41e41437783099c2477819ef6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:04:46 UTC) from {username='oryx', principal='oryx/127.5.83.60@KRBTEST.COM'} at 127.0.0.1:36962
I20260808 04:04:46.823462 15554 tablet_service.cc:1514] Processing DeleteTablet for tablet fa1b13b41e41437783099c2477819ef6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:04:46 UTC) from {username='oryx', principal='oryx/127.5.83.60@KRBTEST.COM'} at 127.0.0.1:42910
I20260808 04:04:46.823853 15753 tablet_service.cc:1514] Processing DeleteTablet for tablet fa1b13b41e41437783099c2477819ef6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:04:46 UTC) from {username='oryx', principal='oryx/127.5.83.60@KRBTEST.COM'} at 127.0.0.1:50904
I20260808 04:04:46.824106 16035 tablet_replica.cc:330] T fa1b13b41e41437783099c2477819ef6 P 82a19f4b641448c384fdc9198ab1143e: stopping tablet replica
I20260808 04:04:46.824347 16035 raft_consensus.cc:2229] T fa1b13b41e41437783099c2477819ef6 P 82a19f4b641448c384fdc9198ab1143e [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:04:46.824993 16035 raft_consensus.cc:2258] T fa1b13b41e41437783099c2477819ef6 P 82a19f4b641448c384fdc9198ab1143e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:04:46.827175 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 15127
I20260808 04:04:46.832424 16035 ts_tablet_manager.cc:1905] T fa1b13b41e41437783099c2477819ef6 P 82a19f4b641448c384fdc9198ab1143e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:04:46.833925 16037 tablet_replica.cc:330] T fa1b13b41e41437783099c2477819ef6 P 4e9480f289294032b1d7d1018594001c: stopping tablet replica
I20260808 04:04:46.834255 16037 raft_consensus.cc:2229] T fa1b13b41e41437783099c2477819ef6 P 4e9480f289294032b1d7d1018594001c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:04:46.834609 16037 raft_consensus.cc:2258] T fa1b13b41e41437783099c2477819ef6 P 4e9480f289294032b1d7d1018594001c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:04:46.835855 16037 ts_tablet_manager.cc:1905] T fa1b13b41e41437783099c2477819ef6 P 4e9480f289294032b1d7d1018594001c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:04:46.839977 16035 ts_tablet_manager.cc:1918] T fa1b13b41e41437783099c2477819ef6 P 82a19f4b641448c384fdc9198ab1143e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:04:46.840539 16036 tablet_replica.cc:330] T fa1b13b41e41437783099c2477819ef6 P 861039c94e114a4c9afc2caeb3282448: stopping tablet replica
I20260808 04:04:46.840739 16036 raft_consensus.cc:2229] T fa1b13b41e41437783099c2477819ef6 P 861039c94e114a4c9afc2caeb3282448 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:04:46.841074 16036 raft_consensus.cc:2258] T fa1b13b41e41437783099c2477819ef6 P 861039c94e114a4c9afc2caeb3282448 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:04:46.842585 16035 log.cc:1196] T fa1b13b41e41437783099c2477819ef6 P 82a19f4b641448c384fdc9198ab1143e: Deleting WAL directory at /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-0/wal/wals/fa1b13b41e41437783099c2477819ef6
I20260808 04:04:46.843129 16035 ts_tablet_manager.cc:1939] T fa1b13b41e41437783099c2477819ef6 P 82a19f4b641448c384fdc9198ab1143e: Deleting consensus metadata
I20260808 04:04:46.843333 16037 ts_tablet_manager.cc:1918] T fa1b13b41e41437783099c2477819ef6 P 4e9480f289294032b1d7d1018594001c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:04:46.844431 16036 ts_tablet_manager.cc:1905] T fa1b13b41e41437783099c2477819ef6 P 861039c94e114a4c9afc2caeb3282448: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:04:46.844440 15152 catalog_manager.cc:4841] TS 82a19f4b641448c384fdc9198ab1143e (127.5.83.1:45257): tablet fa1b13b41e41437783099c2477819ef6 (table test-table [id=0184e889267a4ae2aa275f982453efa2]) successfully deleted
I20260808 04:04:46.844875 16037 log.cc:1196] T fa1b13b41e41437783099c2477819ef6 P 4e9480f289294032b1d7d1018594001c: Deleting WAL directory at /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-2/wal/wals/fa1b13b41e41437783099c2477819ef6
I20260808 04:04:46.845330 16037 ts_tablet_manager.cc:1939] T fa1b13b41e41437783099c2477819ef6 P 4e9480f289294032b1d7d1018594001c: Deleting consensus metadata
I20260808 04:04:46.846398 15150 catalog_manager.cc:4841] TS 4e9480f289294032b1d7d1018594001c (127.5.83.3:46513): tablet fa1b13b41e41437783099c2477819ef6 (table test-table [id=0184e889267a4ae2aa275f982453efa2]) successfully deleted
I20260808 04:04:46.850795 16036 ts_tablet_manager.cc:1918] T fa1b13b41e41437783099c2477819ef6 P 861039c94e114a4c9afc2caeb3282448: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:04:46.850939 16036 log.cc:1196] T fa1b13b41e41437783099c2477819ef6 P 861039c94e114a4c9afc2caeb3282448: Deleting WAL directory at /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161826605655-5452-0/minicluster-data/ts-1/wal/wals/fa1b13b41e41437783099c2477819ef6
I20260808 04:04:46.851313 16036 ts_tablet_manager.cc:1939] T fa1b13b41e41437783099c2477819ef6 P 861039c94e114a4c9afc2caeb3282448: Deleting consensus metadata
I20260808 04:04:46.852275 15153 catalog_manager.cc:4841] TS 861039c94e114a4c9afc2caeb3282448 (127.5.83.2:46681): tablet fa1b13b41e41437783099c2477819ef6 (table test-table [id=0184e889267a4ae2aa275f982453efa2]) successfully deleted
I20260808 04:04:46.856864 15998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:46.828302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36984 (local address 127.5.83.1:45257)
0808 04:04:46.828497 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:46.828504 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:46.836160 (+ 7656us) server_negotiation.cc:407] Connection header received
0808 04:04:46.840385 (+ 4225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:46.840391 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:46.840513 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:46.840610 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:46.842521 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:46.843824 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:46.845687 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:46.846345 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:46.846501 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:46.849752 (+ 3251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:46.849766 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:46.849787 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:46.849827 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:46.853811 (+ 3984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:46.854474 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:46.854483 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:46.854487 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:46.854688 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:46.855063 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:46.855068 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:46.855071 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:46.855586 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:46.855915 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:46.856246 (+ 331us) server_negotiation.cc:299] Negotiation successful
0808 04:04:46.856478 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 04:04:46.857570 15422 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:04:46.958364 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 15127
I20260808 04:04:46.982389 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 15449
I20260808 04:04:46.994949 16001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:46.983118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42920 (local address 127.5.83.2:46681)
0808 04:04:46.983273 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:46.983277 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:46.983313 (+ 36us) server_negotiation.cc:407] Connection header received
0808 04:04:46.983472 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:46.983477 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:46.983540 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:46.983623 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:46.984948 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:46.985674 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:46.986862 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:46.987271 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:46.987395 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:46.989654 (+ 2259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:46.989670 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:46.989685 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:46.989718 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:46.992745 (+ 3027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:46.993327 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:46.993334 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:46.993337 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:46.993466 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:46.993735 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:46.993737 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:46.993739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:46.994087 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:46.994321 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:46.994496 (+ 175us) server_negotiation.cc:299] Negotiation successful
0808 04:04:46.994677 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 04:04:46.995571 15638 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:04:47.090337 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 15449
I20260808 04:04:47.110591 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 15668
I20260808 04:04:47.123854 16004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:47.111311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50908 (local address 127.5.83.3:46513)
0808 04:04:47.111473 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:47.111477 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:47.111563 (+ 86us) server_negotiation.cc:407] Connection header received
0808 04:04:47.111745 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:47.111751 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:47.111818 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:47.111913 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:47.113284 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:47.114058 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:47.115254 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:47.115651 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:47.115851 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:47.118240 (+ 2389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:47.118256 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:47.118272 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:47.118305 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:47.121451 (+ 3146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:47.122025 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:47.122033 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:47.122037 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:47.122165 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:47.122463 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:47.122468 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:47.122471 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:47.122866 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:47.123147 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:47.123281 (+ 134us) server_negotiation.cc:299] Negotiation successful
0808 04:04:47.123493 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 04:04:47.124552 15848 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:04:47.207109 15977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:47.206664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:39395 (local address 127.0.0.1:34382)
0808 04:04:47.206839 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:47.206960 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:39395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260808 04:04:47.208146 15977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:47.207819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:39395 (local address 127.0.0.1:34392)
0808 04:04:47.207967 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:47.208037 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:39395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260808 04:04:47.241781 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 15668
I20260808 04:04:47.267516 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 14714
I20260808 04:04:47.268648 14954 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:04:47.384626 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 14714
I20260808 04:04:47.407769 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 14860
I20260808 04:04:47.408824 15219 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:04:47.528525 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 14860
2026-08-08T04:04:47Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (11432 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:04:47 dist-test-slave-1l3l krb5kdc[16047](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:04:47 dist-test-slave-1l3l krb5kdc[16047](info): set up 2 sockets
Aug 08 04:04:47 dist-test-slave-1l3l krb5kdc[16047](info): commencing operation
krb5kdc: starting...
W20260808 04:04:48.720093 5452 mini_kdc.cc:121] Time spent starting KDC: real 1.140s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/krb5kdc/test-user.keytab.
Aug 08 04:04:48 dist-test-slave-1l3l krb5kdc[16047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161888, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:04:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:04:48Z Disabled control of system clock
WARNING: no policy specified for oryx/127.5.83.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.83.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.83.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/krb5kdc/oryx_127.5.83.62.keytab.
Entry for principal oryx/127.5.83.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/krb5kdc/oryx_127.5.83.62.keytab.
I20260808 04:04:48.930940 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:33481
--webserver_interface=127.5.83.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/krb5kdc/oryx_127.5.83.62.keytab
--principal=oryx/127.5.83.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:44927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.62:33481
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:04:49.246073 16061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:04:49.246341 16061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:04:49.246460 16061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:04:49.253473 16061 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:04:49.253567 16061 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260808 04:04:49.253614 16061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:04:49.253703 16061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:04:49.253763 16061 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:04:49.253845 16061 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:04:49.263239 16061 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:44927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.62:33481
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/krb5kdc/oryx_127.5.83.62.keytab
--principal=oryx/127.5.83.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:33481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:04:49.264417 16061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:04:49.265641 16061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:04:49.276571 16069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:49.276612 16070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:49.276532 16074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:04:49.277678 16061 server_base.cc:1034] running on GCE node
I20260808 04:04:49.278337 16061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:04:49.279563 16061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:04:49.280799 16061 hybrid_clock.cc:648] HybridClock initialized: now 1786161889280736 us; error 52 us; skew 500 ppm
I20260808 04:04:49.281096 16061 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:04:49 dist-test-slave-1l3l krb5kdc[16047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161889, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.83.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:04:49.283950 16061 init.cc:377] Logged in from keytab as oryx/127.5.83.62@KRBTEST.COM (short username oryx)
I20260808 04:04:49.285247 16061 webserver.cc:460] Webserver started at http://127.5.83.62:43553/ using document root <none> and password file <none>
I20260808 04:04:49.285869 16061 fs_manager.cc:359] Metadata directory not provided
I20260808 04:04:49.285967 16061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:04:49.286258 16061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:04:49.289031 16061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/minicluster-data/master-0/data/instance:
uuid: "9f9dc08ed53644aa9bd23b443d2ae405"
format_stamp: "Formatted at 2026-08-08 04:04:49 on dist-test-slave-1l3l"
server_key: "eab4647af04cd0fae186dfc4b7570c0e"
server_key_iv: "6db7165ce74182a5f247b69abd3891af"
server_key_version: "encryptionkey@0"
I20260808 04:04:49.289769 16061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/minicluster-data/master-0/wal/instance:
uuid: "9f9dc08ed53644aa9bd23b443d2ae405"
format_stamp: "Formatted at 2026-08-08 04:04:49 on dist-test-slave-1l3l"
server_key: "eab4647af04cd0fae186dfc4b7570c0e"
server_key_iv: "6db7165ce74182a5f247b69abd3891af"
server_key_version: "encryptionkey@0"
I20260808 04:04:49.294306 16061 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 04:04:49.297439 16085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:49.298589 16061 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 04:04:49.298741 16061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/minicluster-data/master-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/minicluster-data/master-0/wal
uuid: "9f9dc08ed53644aa9bd23b443d2ae405"
format_stamp: "Formatted at 2026-08-08 04:04:49 on dist-test-slave-1l3l"
server_key: "eab4647af04cd0fae186dfc4b7570c0e"
server_key_iv: "6db7165ce74182a5f247b69abd3891af"
server_key_version: "encryptionkey@0"
I20260808 04:04:49.298933 16061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:04:49.314656 16061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:04:49.318092 16061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:04:49.318311 16061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:04:49.337632 16061 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.62:33481
I20260808 04:04:49.337664 16178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.62:33481 every 8 connection(s)
I20260808 04:04:49.338789 16061 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
I20260808 04:04:49.339514 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 16061
I20260808 04:04:49.339733 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/minicluster-data/master-0/wal/instance
I20260808 04:04:49.340446 5452 external_mini_cluster.cc:1428] Setting key c09e4e50da66fad0cbacf5ee9d7d2624
I20260808 04:04:49.342757 16179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:04:49.352588 16179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f9dc08ed53644aa9bd23b443d2ae405: Bootstrap starting.
Aug 08 04:04:49 dist-test-slave-1l3l krb5kdc[16047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161888, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM
I20260808 04:04:49.355679 16179 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f9dc08ed53644aa9bd23b443d2ae405: Neither blocks nor log segments found. Creating new log.
I20260808 04:04:49.356696 16179 log.cc:824] T 00000000000000000000000000000000 P 9f9dc08ed53644aa9bd23b443d2ae405: Log is configured to *not* fsync() on all Append() calls
I20260808 04:04:49.359115 16179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f9dc08ed53644aa9bd23b443d2ae405: No bootstrap required, opened a new log
I20260808 04:04:49.360286 16185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:49.342217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55458 (local address 127.5.83.62:33481)
0808 04:04:49.342958 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:49.342969 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:49.343011 (+ 42us) server_negotiation.cc:407] Connection header received
0808 04:04:49.344616 (+ 1605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:49.344636 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:49.344979 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:49.345310 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:49.346671 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:49.348021 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:49.349202 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:49.349917 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:49.350089 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:49.353787 (+ 3698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:49.353797 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:49.353809 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:49.353846 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:49.356683 (+ 2837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:49.357305 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:49.357311 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:49.357319 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:49.357482 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:49.357844 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:49.357847 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:49.357849 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:49.358295 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:49.358563 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:49.358970 (+ 407us) server_negotiation.cc:299] Negotiation successful
0808 04:04:49.359350 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":330,"threads_started":1}
I20260808 04:04:49.366565 16179 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f9dc08ed53644aa9bd23b443d2ae405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f9dc08ed53644aa9bd23b443d2ae405" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 33481 } }
I20260808 04:04:49.366866 16179 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f9dc08ed53644aa9bd23b443d2ae405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:04:49.366948 16179 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f9dc08ed53644aa9bd23b443d2ae405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f9dc08ed53644aa9bd23b443d2ae405, State: Initialized, Role: FOLLOWER
I20260808 04:04:49.367460 16179 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f9dc08ed53644aa9bd23b443d2ae405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f9dc08ed53644aa9bd23b443d2ae405" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 33481 } }
I20260808 04:04:49.367668 16179 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f9dc08ed53644aa9bd23b443d2ae405 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:04:49.367847 16179 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f9dc08ed53644aa9bd23b443d2ae405 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:04:49.367997 16179 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f9dc08ed53644aa9bd23b443d2ae405 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:04:49.370180 16179 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f9dc08ed53644aa9bd23b443d2ae405 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f9dc08ed53644aa9bd23b443d2ae405" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 33481 } }
I20260808 04:04:49.370639 16179 leader_election.cc:302] T 00000000000000000000000000000000 P 9f9dc08ed53644aa9bd23b443d2ae405 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f9dc08ed53644aa9bd23b443d2ae405; no voters:
I20260808 04:04:49.371201 16179 leader_election.cc:288] T 00000000000000000000000000000000 P 9f9dc08ed53644aa9bd23b443d2ae405 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:04:49.371326 16190 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f9dc08ed53644aa9bd23b443d2ae405 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:04:49.372010 16190 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f9dc08ed53644aa9bd23b443d2ae405 [term 1 LEADER]: Becoming Leader. State: Replica: 9f9dc08ed53644aa9bd23b443d2ae405, State: Running, Role: LEADER
I20260808 04:04:49.372570 16190 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f9dc08ed53644aa9bd23b443d2ae405 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f9dc08ed53644aa9bd23b443d2ae405" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 33481 } }
I20260808 04:04:49.373011 16179 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f9dc08ed53644aa9bd23b443d2ae405 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:04:49.375387 16194 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f9dc08ed53644aa9bd23b443d2ae405 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f9dc08ed53644aa9bd23b443d2ae405. Latest consensus state: current_term: 1 leader_uuid: "9f9dc08ed53644aa9bd23b443d2ae405" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f9dc08ed53644aa9bd23b443d2ae405" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 33481 } } }
I20260808 04:04:49.375699 16194 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f9dc08ed53644aa9bd23b443d2ae405 [sys.catalog]: This master's current role is: LEADER
I20260808 04:04:49.374996 16191 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f9dc08ed53644aa9bd23b443d2ae405 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f9dc08ed53644aa9bd23b443d2ae405" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f9dc08ed53644aa9bd23b443d2ae405" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 33481 } } }
I20260808 04:04:49.376283 16191 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f9dc08ed53644aa9bd23b443d2ae405 [sys.catalog]: This master's current role is: LEADER
I20260808 04:04:49.377890 16208 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:04:49.382726 16208 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:04:49.390733 16208 catalog_manager.cc:1347] Generated new cluster ID: 65f29efeddae410b992238407a86e7c2
I20260808 04:04:49.390868 16208 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:04:49.435453 16208 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:04:49.436802 16208 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:04:49.444566 16208 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f9dc08ed53644aa9bd23b443d2ae405: Generated new TSK 0
I20260808 04:04:49.445324 16208 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.5.83.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.83.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.83.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/krb5kdc/oryx_127.5.83.1.keytab.
Entry for principal oryx/127.5.83.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/krb5kdc/oryx_127.5.83.1.keytab.
I20260808 04:04:49.513212 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.1:0
--local_ip_for_outbound_sockets=127.5.83.1
--webserver_interface=127.5.83.1
--webserver_port=0
--tserver_master_addrs=127.5.83.62:33481
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/krb5kdc/oryx_127.5.83.1.keytab
--principal=oryx/127.5.83.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:44927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:04:49.831737 16224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:04:49.831991 16224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:04:49.832125 16224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:04:49.839095 16224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:04:49.839195 16224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:04:49.839314 16224 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 04:04:49.839417 16224 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.1
I20260808 04:04:49.848796 16224 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:44927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/krb5kdc/oryx_127.5.83.1.keytab
--principal=oryx/127.5.83.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.83.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.83.62:33481
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.1
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:04:49.849984 16224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:04:49.851240 16224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:04:49.863617 16235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:49.863801 16232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:49.863780 16231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:04:49.864614 16224 server_base.cc:1034] running on GCE node
I20260808 04:04:49.865188 16224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:04:49.866012 16224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:04:49.867224 16224 hybrid_clock.cc:648] HybridClock initialized: now 1786161889867167 us; error 32 us; skew 500 ppm
I20260808 04:04:49.867486 16224 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:04:49 dist-test-slave-1l3l krb5kdc[16047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161889, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.83.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:04:49.870558 16224 init.cc:377] Logged in from keytab as oryx/127.5.83.1@KRBTEST.COM (short username oryx)
I20260808 04:04:49.871955 16224 webserver.cc:460] Webserver started at http://127.5.83.1:37203/ using document root <none> and password file <none>
I20260808 04:04:49.872644 16224 fs_manager.cc:359] Metadata directory not provided
I20260808 04:04:49.872750 16224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:04:49.873050 16224 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:04:49.875877 16224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/minicluster-data/ts-0/data/instance:
uuid: "2013510f40c14486a71a3f2d3668d06a"
format_stamp: "Formatted at 2026-08-08 04:04:49 on dist-test-slave-1l3l"
server_key: "9f0a08a11adc6b9edc7a9523f642cd98"
server_key_iv: "78416a0b6508ed87bf26020fdd292c53"
server_key_version: "encryptionkey@0"
I20260808 04:04:49.876668 16224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/minicluster-data/ts-0/wal/instance:
uuid: "2013510f40c14486a71a3f2d3668d06a"
format_stamp: "Formatted at 2026-08-08 04:04:49 on dist-test-slave-1l3l"
server_key: "9f0a08a11adc6b9edc7a9523f642cd98"
server_key_iv: "78416a0b6508ed87bf26020fdd292c53"
server_key_version: "encryptionkey@0"
I20260808 04:04:49.881425 16224 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 04:04:49.884684 16243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:49.885845 16224 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:04:49.885991 16224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/minicluster-data/ts-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/minicluster-data/ts-0/wal
uuid: "2013510f40c14486a71a3f2d3668d06a"
format_stamp: "Formatted at 2026-08-08 04:04:49 on dist-test-slave-1l3l"
server_key: "9f0a08a11adc6b9edc7a9523f642cd98"
server_key_iv: "78416a0b6508ed87bf26020fdd292c53"
server_key_version: "encryptionkey@0"
I20260808 04:04:49.886195 16224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:04:49.904615 16224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:04:49.908144 16224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:04:49.908383 16224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:04:49.911095 16224 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:04:49.911237 16224 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260808 04:04:49.911353 16224 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:04:49.911417 16224 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 04:04:49 dist-test-slave-1l3l krb5kdc[16047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161889, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.83.1@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM
I20260808 04:04:49.932529 16254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:49.911558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.1:51539 (local address 127.5.83.62:33481)
0808 04:04:49.912305 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:49.912311 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:49.913309 (+ 998us) server_negotiation.cc:407] Connection header received
0808 04:04:49.915263 (+ 1954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:49.915270 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:49.915377 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:49.915508 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:49.917379 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:49.918477 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:49.920145 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:49.920662 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:49.920778 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:49.925105 (+ 4327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:49.925111 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:49.925114 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:49.925149 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:49.928284 (+ 3135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:49.929142 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:49.929148 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:49.929152 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:49.929304 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:49.929959 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:49.929964 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:49.929968 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:49.930370 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:49.930520 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:49.931930 (+ 1410us) server_negotiation.cc:299] Negotiation successful
0808 04:04:49.932147 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":513,"threads_started":1}
I20260808 04:04:49.933121 16255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:49.912286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:33481 (local address 127.5.83.1:51539)
0808 04:04:49.913130 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:49.913173 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:49.915005 (+ 1832us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:49.915653 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:49.915660 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:49.916346 (+ 686us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:04:49.917126 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:49.917149 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:49.918649 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:49.918655 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:49.920006 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:49.920018 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:49.921090 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:49.921098 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:49.921345 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:49.922100 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:04:49.922135 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:04:49.924868 (+ 2733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:04:49.928420 (+ 3552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:49.928428 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:49.928442 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:49.929024 (+ 582us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:49.929444 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:49.929452 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:49.929457 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:49.929779 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:49.930535 (+ 756us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:04:49.930542 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:04:49.930945 (+ 403us) client_negotiation.cc:769] Sending connection context
0808 04:04:49.931277 (+ 332us) client_negotiation.cc:240] Negotiation successful
0808 04:04:49.931781 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":376,"threads_started":1}
I20260808 04:04:49.951768 16254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:49.938955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.1:50385 (local address 127.5.83.62:33481)
0808 04:04:49.939131 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:49.939136 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:49.939769 (+ 633us) server_negotiation.cc:407] Connection header received
0808 04:04:49.939949 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:49.939954 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:49.940029 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:49.940123 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:49.941375 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:49.942321 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:49.944437 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:49.944946 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:49.945063 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:49.947041 (+ 1978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:49.947046 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:49.947050 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:49.947079 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:49.949394 (+ 2315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:49.949991 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:49.949996 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:49.949999 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:49.950118 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:49.950453 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:49.950458 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:49.950462 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:49.950703 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:49.950829 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:49.951225 (+ 396us) server_negotiation.cc:299] Negotiation successful
0808 04:04:49.951431 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 04:04:49.951870 16362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:49.938961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:33481 (local address 127.5.83.1:50385)
0808 04:04:49.939621 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:49.939645 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:49.939823 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:49.940196 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:49.940201 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:49.940802 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:04:49.941244 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:49.941258 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:49.942452 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:49.942457 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:49.944037 (+ 1580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:49.944054 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:49.945128 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:49.945136 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:49.945303 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:49.945852 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:04:49.945872 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:04:49.946860 (+ 988us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:04:49.949523 (+ 2663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:49.949528 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:49.949532 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:49.949879 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:49.950220 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:49.950225 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:49.950229 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:49.950342 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:49.950811 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:04:49.950816 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:04:49.950983 (+ 167us) client_negotiation.cc:769] Sending connection context
0808 04:04:49.951109 (+ 126us) client_negotiation.cc:240] Negotiation successful
0808 04:04:49.951385 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":353,"threads_started":1}
I20260808 04:04:49.958129 16224 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.1:35837
I20260808 04:04:49.958170 16435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.1:35837 every 8 connection(s)
I20260808 04:04:49.959203 16224 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
I20260808 04:04:49.961068 16437 heartbeater.cc:347] Connected to a master server at 127.5.83.62:33481
I20260808 04:04:49.961263 16437 heartbeater.cc:464] Registering TS with master...
I20260808 04:04:49.961850 16437 heartbeater.cc:511] Master 127.5.83.62:33481 requested a full tablet report, sending...
I20260808 04:04:49.963507 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 16224
I20260808 04:04:49.963604 16112 ts_manager.cc:194] Registered new tserver with Master: 2013510f40c14486a71a3f2d3668d06a (127.5.83.1:35837)
I20260808 04:04:49.963676 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/minicluster-data/ts-0/wal/instance
I20260808 04:04:49.964316 5452 external_mini_cluster.cc:1428] Setting key b520228b30f641b4f650bf09dc68e7b2
I20260808 04:04:49.965075 16112 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.83.1@KRBTEST.COM'} at 127.5.83.1:51539
WARNING: no policy specified for oryx/127.5.83.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.83.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.83.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/krb5kdc/oryx_127.5.83.2.keytab.
Entry for principal oryx/127.5.83.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/krb5kdc/oryx_127.5.83.2.keytab.
I20260808 04:04:50.012907 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.2:0
--local_ip_for_outbound_sockets=127.5.83.2
--webserver_interface=127.5.83.2
--webserver_port=0
--tserver_master_addrs=127.5.83.62:33481
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/krb5kdc/oryx_127.5.83.2.keytab
--principal=oryx/127.5.83.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:44927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:04:50.327862 16443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:04:50.328109 16443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:04:50.328202 16443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:04:50.335173 16443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:04:50.335273 16443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:04:50.335372 16443 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 04:04:50.335458 16443 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.2
I20260808 04:04:50.344908 16443 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:44927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/krb5kdc/oryx_127.5.83.2.keytab
--principal=oryx/127.5.83.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.83.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.83.62:33481
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.2
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:04:50.346000 16443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:04:50.347198 16443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:04:50.359324 16452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:50.359373 16453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:50.359426 16456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:04:50.360272 16443 server_base.cc:1034] running on GCE node
I20260808 04:04:50.360849 16443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:04:50.361666 16443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:04:50.362893 16443 hybrid_clock.cc:648] HybridClock initialized: now 1786161890362838 us; error 35 us; skew 500 ppm
I20260808 04:04:50.363166 16443 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:04:50 dist-test-slave-1l3l krb5kdc[16047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161890, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.83.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:04:50.366194 16443 init.cc:377] Logged in from keytab as oryx/127.5.83.2@KRBTEST.COM (short username oryx)
I20260808 04:04:50.367579 16443 webserver.cc:460] Webserver started at http://127.5.83.2:43319/ using document root <none> and password file <none>
I20260808 04:04:50.368237 16443 fs_manager.cc:359] Metadata directory not provided
I20260808 04:04:50.368340 16443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:04:50.368649 16443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:04:50.371372 16443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/minicluster-data/ts-1/data/instance:
uuid: "442fa005e54743748ca0af7671c4a79a"
format_stamp: "Formatted at 2026-08-08 04:04:50 on dist-test-slave-1l3l"
server_key: "8b1d0c3bc07f9652b80a910d6fd1b093"
server_key_iv: "47271471a4ec1cd044848d194c6dbc85"
server_key_version: "encryptionkey@0"
I20260808 04:04:50.372164 16443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/minicluster-data/ts-1/wal/instance:
uuid: "442fa005e54743748ca0af7671c4a79a"
format_stamp: "Formatted at 2026-08-08 04:04:50 on dist-test-slave-1l3l"
server_key: "8b1d0c3bc07f9652b80a910d6fd1b093"
server_key_iv: "47271471a4ec1cd044848d194c6dbc85"
server_key_version: "encryptionkey@0"
I20260808 04:04:50.377218 16443 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260808 04:04:50.380548 16469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:50.381800 16443 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:04:50.381956 16443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/minicluster-data/ts-1/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/minicluster-data/ts-1/wal
uuid: "442fa005e54743748ca0af7671c4a79a"
format_stamp: "Formatted at 2026-08-08 04:04:50 on dist-test-slave-1l3l"
server_key: "8b1d0c3bc07f9652b80a910d6fd1b093"
server_key_iv: "47271471a4ec1cd044848d194c6dbc85"
server_key_version: "encryptionkey@0"
I20260808 04:04:50.382154 16443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:04:50.403817 16443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:04:50.407547 16443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:04:50.407823 16443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:04:50.410426 16443 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:04:50.410537 16443 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:50.410681 16443 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:04:50.410755 16443 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 04:04:50 dist-test-slave-1l3l krb5kdc[16047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161890, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.83.2@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM
I20260808 04:04:50.432519 16254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:50.410804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.2:48379 (local address 127.5.83.62:33481)
0808 04:04:50.410994 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:50.411000 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:50.412442 (+ 1442us) server_negotiation.cc:407] Connection header received
0808 04:04:50.414486 (+ 2044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:50.414493 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:50.414584 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:50.414714 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:50.416786 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:50.417973 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:50.420324 (+ 2351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:50.420743 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:50.420863 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:50.425486 (+ 4623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:50.425493 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:50.425498 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:50.425543 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:50.428861 (+ 3318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:50.429664 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:50.429669 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:50.429672 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:50.429829 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:50.430498 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:50.430503 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:50.430507 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:50.430799 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:50.430920 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:50.432096 (+ 1176us) server_negotiation.cc:299] Negotiation successful
0808 04:04:50.432270 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260808 04:04:50.434062 16486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:50.411437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:33481 (local address 127.5.83.2:48379)
0808 04:04:50.412251 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:50.412318 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:50.414138 (+ 1820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:50.414984 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:50.414994 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:50.415788 (+ 794us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:04:50.416570 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:50.416596 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:50.418138 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:50.418146 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:50.420179 (+ 2033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:50.420195 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:50.421214 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:50.421220 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:50.421445 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:50.422130 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:04:50.422179 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:04:50.425183 (+ 3004us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:04:50.429034 (+ 3851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:50.429043 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:50.429058 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:50.429499 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:50.429996 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:50.430006 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:50.430010 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:50.430343 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:50.430935 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:04:50.430943 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:04:50.431505 (+ 562us) client_negotiation.cc:769] Sending connection context
0808 04:04:50.431986 (+ 481us) client_negotiation.cc:240] Negotiation successful
0808 04:04:50.432552 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":383,"threads_started":1}
I20260808 04:04:50.451664 16254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:50.439363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.2:36193 (local address 127.5.83.62:33481)
0808 04:04:50.439501 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:50.439505 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:50.440146 (+ 641us) server_negotiation.cc:407] Connection header received
0808 04:04:50.440300 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:50.440303 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:50.440358 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:50.440431 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:50.441294 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:50.442004 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:50.443377 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:50.443807 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:50.443946 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:50.446092 (+ 2146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:50.446096 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:50.446100 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:50.446137 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:50.448828 (+ 2691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:50.449602 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:50.449608 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:50.449612 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:50.449772 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:50.450269 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:50.450274 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:50.450277 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:50.450649 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:50.450800 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:50.451265 (+ 465us) server_negotiation.cc:299] Negotiation successful
0808 04:04:50.451443 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 04:04:50.452061 16576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:50.439420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:33481 (local address 127.5.83.2:36193)
0808 04:04:50.440056 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:50.440070 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:50.440189 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:50.440466 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:50.440471 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:50.440900 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:04:50.441165 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:50.441174 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:50.442144 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:50.442148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:50.443251 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:50.443264 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:50.444041 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:50.444053 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:50.444287 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:50.444980 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:04:50.445003 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:04:50.445929 (+ 926us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:04:50.448949 (+ 3020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:50.448954 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:50.448957 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:50.449466 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:50.449947 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:50.449952 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:50.449960 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:50.450141 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:50.450787 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:04:50.450792 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:04:50.451014 (+ 222us) client_negotiation.cc:769] Sending connection context
0808 04:04:50.451198 (+ 184us) client_negotiation.cc:240] Negotiation successful
0808 04:04:50.451534 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":384,"threads_started":1}
I20260808 04:04:50.463901 16443 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.2:39807
I20260808 04:04:50.463923 16647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.2:39807 every 8 connection(s)
I20260808 04:04:50.465052 16443 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
I20260808 04:04:50.466745 16648 heartbeater.cc:347] Connected to a master server at 127.5.83.62:33481
I20260808 04:04:50.466944 16648 heartbeater.cc:464] Registering TS with master...
I20260808 04:04:50.467483 16648 heartbeater.cc:511] Master 127.5.83.62:33481 requested a full tablet report, sending...
I20260808 04:04:50.468708 16108 ts_manager.cc:194] Registered new tserver with Master: 442fa005e54743748ca0af7671c4a79a (127.5.83.2:39807)
I20260808 04:04:50.469508 16108 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.83.2@KRBTEST.COM'} at 127.5.83.2:48379
I20260808 04:04:50.473603 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 16443
I20260808 04:04:50.473757 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/minicluster-data/ts-1/wal/instance
I20260808 04:04:50.474188 5452 external_mini_cluster.cc:1428] Setting key a1372611ea55bc789220bb2745fb9ab9
WARNING: no policy specified for oryx/127.5.83.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.83.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.83.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/krb5kdc/oryx_127.5.83.3.keytab.
Entry for principal oryx/127.5.83.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/krb5kdc/oryx_127.5.83.3.keytab.
I20260808 04:04:50.520267 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.3:0
--local_ip_for_outbound_sockets=127.5.83.3
--webserver_interface=127.5.83.3
--webserver_port=0
--tserver_master_addrs=127.5.83.62:33481
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/krb5kdc/oryx_127.5.83.3.keytab
--principal=oryx/127.5.83.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:44927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:04:50.833945 16652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:04:50.834236 16652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:04:50.834385 16652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:04:50.841820 16652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:04:50.841938 16652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:04:50.842073 16652 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 04:04:50.842187 16652 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.3
I20260808 04:04:50.851742 16652 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:44927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/krb5kdc/oryx_127.5.83.3.keytab
--principal=oryx/127.5.83.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.83.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.83.62:33481
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.3
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:04:50.853011 16652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:04:50.854314 16652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:04:50.866879 16659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:50.866839 16660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:50.866910 16666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:04:50.867913 16652 server_base.cc:1034] running on GCE node
I20260808 04:04:50.868530 16652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:04:50.869336 16652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:04:50.870555 16652 hybrid_clock.cc:648] HybridClock initialized: now 1786161890870490 us; error 38 us; skew 500 ppm
I20260808 04:04:50.870822 16652 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:04:50 dist-test-slave-1l3l krb5kdc[16047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161890, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.83.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:04:50.873888 16652 init.cc:377] Logged in from keytab as oryx/127.5.83.3@KRBTEST.COM (short username oryx)
I20260808 04:04:50.875267 16652 webserver.cc:460] Webserver started at http://127.5.83.3:39407/ using document root <none> and password file <none>
I20260808 04:04:50.876015 16652 fs_manager.cc:359] Metadata directory not provided
I20260808 04:04:50.876163 16652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:04:50.876539 16652 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:04:50.879546 16652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/minicluster-data/ts-2/data/instance:
uuid: "c230911ad282421cabd56a069e80976f"
format_stamp: "Formatted at 2026-08-08 04:04:50 on dist-test-slave-1l3l"
server_key: "51e799e4ab0a0adf5a7c7f1ea0a8363d"
server_key_iv: "2189038b25eff9b0624395c07328fab8"
server_key_version: "encryptionkey@0"
I20260808 04:04:50.880363 16652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/minicluster-data/ts-2/wal/instance:
uuid: "c230911ad282421cabd56a069e80976f"
format_stamp: "Formatted at 2026-08-08 04:04:50 on dist-test-slave-1l3l"
server_key: "51e799e4ab0a0adf5a7c7f1ea0a8363d"
server_key_iv: "2189038b25eff9b0624395c07328fab8"
server_key_version: "encryptionkey@0"
I20260808 04:04:50.885133 16652 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 04:04:50.888451 16673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:50.889726 16652 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:04:50.889881 16652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/minicluster-data/ts-2/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/minicluster-data/ts-2/wal
uuid: "c230911ad282421cabd56a069e80976f"
format_stamp: "Formatted at 2026-08-08 04:04:50 on dist-test-slave-1l3l"
server_key: "51e799e4ab0a0adf5a7c7f1ea0a8363d"
server_key_iv: "2189038b25eff9b0624395c07328fab8"
server_key_version: "encryptionkey@0"
I20260808 04:04:50.890089 16652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:04:50.903875 16652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:04:50.907435 16652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:04:50.907641 16652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:04:50.910108 16652 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:04:50.910183 16652 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:50.910300 16652 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:04:50.910355 16652 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 04:04:50 dist-test-slave-1l3l krb5kdc[16047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161890, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.83.3@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM
I20260808 04:04:50.933897 16254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:50.910614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.3:44591 (local address 127.5.83.62:33481)
0808 04:04:50.910799 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:50.910805 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:50.912167 (+ 1362us) server_negotiation.cc:407] Connection header received
0808 04:04:50.914134 (+ 1967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:50.914140 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:50.914230 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:50.914320 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:50.916353 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:50.917385 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:50.920177 (+ 2792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:50.920704 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:50.920838 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:50.925552 (+ 4714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:50.925558 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:50.925561 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:50.925596 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:50.929102 (+ 3506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:50.930054 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:50.930061 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:50.930064 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:50.930250 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:50.930825 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:50.930831 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:50.930835 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:50.931230 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:50.931393 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:50.933468 (+ 2075us) server_negotiation.cc:299] Negotiation successful
0808 04:04:50.933646 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 04:04:50.934310 16685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:50.911130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:33481 (local address 127.5.83.3:44591)
0808 04:04:50.911979 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:50.912035 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:50.913855 (+ 1820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:50.914571 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:50.914579 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:50.915254 (+ 675us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:04:50.916165 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:50.916195 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:50.917542 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:50.917550 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:50.920013 (+ 2463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:50.920031 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:50.920994 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:50.921003 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:50.921310 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:50.922192 (+ 882us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:04:50.922224 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:04:50.925325 (+ 3101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:04:50.929238 (+ 3913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:50.929248 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:50.929267 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:50.929902 (+ 635us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:50.930396 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:50.930407 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:50.930410 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:50.930676 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:50.931404 (+ 728us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:04:50.931409 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:04:50.931804 (+ 395us) client_negotiation.cc:769] Sending connection context
0808 04:04:50.932105 (+ 301us) client_negotiation.cc:240] Negotiation successful
0808 04:04:50.932672 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":352,"threads_started":1}
I20260808 04:04:50.954200 16254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:50.939584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.3:35429 (local address 127.5.83.62:33481)
0808 04:04:50.939757 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:50.939762 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:50.940758 (+ 996us) server_negotiation.cc:407] Connection header received
0808 04:04:50.940925 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:50.940931 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:50.941007 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:50.941100 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:50.942600 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:50.943489 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:50.945566 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:50.945947 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:50.946073 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:50.948842 (+ 2769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:50.948847 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:50.948850 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:50.948880 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:50.951335 (+ 2455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:50.952203 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:50.952208 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:50.952212 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:50.952424 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:50.952815 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:50.952820 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:50.952824 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:50.953095 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:50.953222 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:50.953643 (+ 421us) server_negotiation.cc:299] Negotiation successful
0808 04:04:50.953859 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 04:04:50.954258 16791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:50.939642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:33481 (local address 127.5.83.3:35429)
0808 04:04:50.940663 (+ 1021us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:50.940679 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:50.940801 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:50.941161 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:50.941166 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:50.941920 (+ 754us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:04:50.942421 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:50.942440 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:50.943623 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:50.943629 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:50.945431 (+ 1802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:50.945450 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:50.946190 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:50.946201 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:50.946399 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:50.947092 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:04:50.947118 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:04:50.948644 (+ 1526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:04:50.951459 (+ 2815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:50.951465 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:50.951469 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:50.952064 (+ 595us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:50.952568 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:50.952573 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:50.952581 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:50.952700 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:50.953204 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:04:50.953209 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:04:50.953372 (+ 163us) client_negotiation.cc:769] Sending connection context
0808 04:04:50.953489 (+ 117us) client_negotiation.cc:240] Negotiation successful
0808 04:04:50.953796 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":896,"thread_start_us":648,"threads_started":1}
I20260808 04:04:50.960610 16652 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.3:43655
I20260808 04:04:50.960628 16862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.3:43655 every 8 connection(s)
I20260808 04:04:50.962042 16652 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
I20260808 04:04:50.964046 16863 heartbeater.cc:347] Connected to a master server at 127.5.83.62:33481
I20260808 04:04:50.964264 16863 heartbeater.cc:464] Registering TS with master...
I20260808 04:04:50.964946 16863 heartbeater.cc:511] Master 127.5.83.62:33481 requested a full tablet report, sending...
I20260808 04:04:50.966084 16108 ts_manager.cc:194] Registered new tserver with Master: c230911ad282421cabd56a069e80976f (127.5.83.3:43655)
I20260808 04:04:50.966948 16108 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.83.3@KRBTEST.COM'} at 127.5.83.3:44591
I20260808 04:04:50.967489 16437 heartbeater.cc:503] Master 127.5.83.62:33481 was elected leader, sending a full tablet report...
I20260808 04:04:50.970541 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 16652
I20260808 04:04:50.970691 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/minicluster-data/ts-2/wal/instance
I20260808 04:04:50.971107 5452 external_mini_cluster.cc:1428] Setting key 7bcdb3ce812020f5705655348a821c17
I20260808 04:04:50.973865 5452 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 04:04:51 dist-test-slave-1l3l krb5kdc[16047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161891, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 04:04:51 dist-test-slave-1l3l krb5kdc[16047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:04:51 dist-test-slave-1l3l krb5kdc[16047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
W20260808 04:04:51.028471 16254 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:04:51.017041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55470 (local address 127.5.83.62:33481)
0808 04:04:51.017252 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:51.017259 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:51.017354 (+ 95us) server_negotiation.cc:407] Connection header received
0808 04:04:51.017560 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:51.017564 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:51.017642 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:51.017730 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:51.019092 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:51.019851 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:51.021106 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:51.021520 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:51.021629 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:51.028164 (+ 6535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55470: BlockingRecv error: recv got EOF from 127.0.0.1:55470 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Aug 08 04:04:51 dist-test-slave-1l3l krb5kdc[16047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161891, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM
I20260808 04:04:51.046518 16254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:51.032743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55474 (local address 127.5.83.62:33481)
0808 04:04:51.032922 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:51.032928 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:51.033277 (+ 349us) server_negotiation.cc:407] Connection header received
0808 04:04:51.033455 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:51.033461 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:51.033543 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:51.033628 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:51.034860 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:51.035733 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:51.037065 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:51.037504 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:51.037615 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:51.041484 (+ 3869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:51.041490 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:51.041493 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:51.041528 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:51.044117 (+ 2589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:51.044728 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:51.044734 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:51.044737 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:51.044863 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:51.045204 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:51.045207 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:51.045209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:51.045481 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:51.045751 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:51.045988 (+ 237us) server_negotiation.cc:299] Negotiation successful
0808 04:04:51.046205 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 04:04:51.050269 16108 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55474:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:04:51.056984 16108 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:04:51.092424 16890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:51.069253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34674 (local address 127.5.83.2:39807)
0808 04:04:51.084222 (+ 14969us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:51.084231 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:51.084261 (+ 30us) server_negotiation.cc:407] Connection header received
0808 04:04:51.084372 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:51.084379 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:51.084626 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:51.085735 (+ 1109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:04:51.085939 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:51.087129 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:51.089905 (+ 2776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:51.090983 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:51.091102 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:51.091883 (+ 781us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:51.091958 (+ 75us) server_negotiation.cc:299] Negotiation successful
0808 04:04:51.092121 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14819,"thread_start_us":14709,"threads_started":1}
I20260808 04:04:51.092697 16895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:51.068964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.2:39807 (local address 127.0.0.1:34674)
0808 04:04:51.083182 (+ 14218us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:51.083202 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:51.083336 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:51.084827 (+ 1491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:51.084831 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:51.084853 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:04:51.085504 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:51.085519 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:51.087255 (+ 1736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:51.087260 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:51.089759 (+ 2499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:51.089772 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:51.091166 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:51.091172 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:51.091347 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:51.091399 (+ 52us) client_negotiation.cc:769] Sending connection context
0808 04:04:51.091522 (+ 123us) client_negotiation.cc:240] Negotiation successful
0808 04:04:51.092348 (+ 826us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14098,"mutex_wait_us":125,"thread_start_us":15348,"threads_started":1}
I20260808 04:04:51.092944 16892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:51.069293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33308 (local address 127.5.83.1:35837)
0808 04:04:51.083906 (+ 14613us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:51.083914 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:51.083949 (+ 35us) server_negotiation.cc:407] Connection header received
0808 04:04:51.084060 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:51.084069 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:51.084346 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:51.084555 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:04:51.085713 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:51.087038 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:51.090119 (+ 3081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:51.091605 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:51.092205 (+ 600us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:51.092411 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:51.092480 (+ 69us) server_negotiation.cc:299] Negotiation successful
0808 04:04:51.092632 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14435,"thread_start_us":14272,"threads_started":1}
I20260808 04:04:51.093638 16887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:51.068825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:35837 (local address 127.0.0.1:33308)
0808 04:04:51.081862 (+ 13037us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:51.081931 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:51.082170 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:51.084810 (+ 2640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:51.084817 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:51.084844 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:04:51.085524 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:51.085536 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:51.087172 (+ 1636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:51.087176 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:51.089893 (+ 2717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:51.089911 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:51.091862 (+ 1951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:51.091870 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:51.092015 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:51.092048 (+ 33us) client_negotiation.cc:769] Sending connection context
0808 04:04:51.092118 (+ 70us) client_negotiation.cc:240] Negotiation successful
0808 04:04:51.093233 (+ 1115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12718,"mutex_wait_us":1137,"thread_start_us":12541,"threads_started":1}
I20260808 04:04:51.094399 16888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:51.070196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.3:43655 (local address 127.0.0.1:41412)
0808 04:04:51.084572 (+ 14376us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:51.084589 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:51.084804 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:51.085783 (+ 979us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:51.085787 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:51.085809 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:04:51.086169 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:51.086181 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:51.088222 (+ 2041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:51.088227 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:51.090355 (+ 2128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:51.090368 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:51.092444 (+ 2076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:51.092451 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:51.093795 (+ 1344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:51.093827 (+ 32us) client_negotiation.cc:769] Sending connection context
0808 04:04:51.093927 (+ 100us) client_negotiation.cc:240] Negotiation successful
0808 04:04:51.094048 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14264,"mutex_wait_us":2,"thread_start_us":12698,"threads_started":1}
I20260808 04:04:51.094592 16898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:51.070938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41412 (local address 127.5.83.3:43655)
0808 04:04:51.082739 (+ 11801us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:51.082750 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:51.084699 (+ 1949us) server_negotiation.cc:407] Connection header received
0808 04:04:51.085015 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:51.085022 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:51.085246 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:51.085408 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:04:51.086435 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:51.088062 (+ 1627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:51.090535 (+ 2473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:51.091960 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:51.092076 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:51.092297 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:51.094095 (+ 1798us) server_negotiation.cc:299] Negotiation successful
0808 04:04:51.094256 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11629,"thread_start_us":11448,"threads_started":1}
I20260808 04:04:51.096710 16550 tablet_service.cc:1467] Processing CreateTablet for tablet 6f3968a617ee4c189de347e185fae427 (DEFAULT_TABLE table=test-table [id=83d5eecc394644fea0b1462545912a87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:04:51.097873 16550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f3968a617ee4c189de347e185fae427. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:04:51.097400 16313 tablet_service.cc:1467] Processing CreateTablet for tablet 6f3968a617ee4c189de347e185fae427 (DEFAULT_TABLE table=test-table [id=83d5eecc394644fea0b1462545912a87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:04:51.097889 16752 tablet_service.cc:1467] Processing CreateTablet for tablet 6f3968a617ee4c189de347e185fae427 (DEFAULT_TABLE table=test-table [id=83d5eecc394644fea0b1462545912a87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:04:51.098721 16313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f3968a617ee4c189de347e185fae427. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:04:51.099241 16752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f3968a617ee4c189de347e185fae427. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:04:51.109247 16899 tablet_bootstrap.cc:492] T 6f3968a617ee4c189de347e185fae427 P 442fa005e54743748ca0af7671c4a79a: Bootstrap starting.
I20260808 04:04:51.112152 16899 tablet_bootstrap.cc:654] T 6f3968a617ee4c189de347e185fae427 P 442fa005e54743748ca0af7671c4a79a: Neither blocks nor log segments found. Creating new log.
I20260808 04:04:51.112874 16900 tablet_bootstrap.cc:492] T 6f3968a617ee4c189de347e185fae427 P c230911ad282421cabd56a069e80976f: Bootstrap starting.
I20260808 04:04:51.113929 16899 log.cc:824] T 6f3968a617ee4c189de347e185fae427 P 442fa005e54743748ca0af7671c4a79a: Log is configured to *not* fsync() on all Append() calls
I20260808 04:04:51.114420 16902 tablet_bootstrap.cc:492] T 6f3968a617ee4c189de347e185fae427 P 2013510f40c14486a71a3f2d3668d06a: Bootstrap starting.
I20260808 04:04:51.116689 16899 tablet_bootstrap.cc:492] T 6f3968a617ee4c189de347e185fae427 P 442fa005e54743748ca0af7671c4a79a: No bootstrap required, opened a new log
I20260808 04:04:51.116930 16899 ts_tablet_manager.cc:1397] T 6f3968a617ee4c189de347e185fae427 P 442fa005e54743748ca0af7671c4a79a: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260808 04:04:51.117640 16900 tablet_bootstrap.cc:654] T 6f3968a617ee4c189de347e185fae427 P c230911ad282421cabd56a069e80976f: Neither blocks nor log segments found. Creating new log.
I20260808 04:04:51.117636 16902 tablet_bootstrap.cc:654] T 6f3968a617ee4c189de347e185fae427 P 2013510f40c14486a71a3f2d3668d06a: Neither blocks nor log segments found. Creating new log.
I20260808 04:04:51.118837 16900 log.cc:824] T 6f3968a617ee4c189de347e185fae427 P c230911ad282421cabd56a069e80976f: Log is configured to *not* fsync() on all Append() calls
I20260808 04:04:51.118839 16902 log.cc:824] T 6f3968a617ee4c189de347e185fae427 P 2013510f40c14486a71a3f2d3668d06a: Log is configured to *not* fsync() on all Append() calls
I20260808 04:04:51.121644 16902 tablet_bootstrap.cc:492] T 6f3968a617ee4c189de347e185fae427 P 2013510f40c14486a71a3f2d3668d06a: No bootstrap required, opened a new log
I20260808 04:04:51.121879 16900 tablet_bootstrap.cc:492] T 6f3968a617ee4c189de347e185fae427 P c230911ad282421cabd56a069e80976f: No bootstrap required, opened a new log
I20260808 04:04:51.121922 16902 ts_tablet_manager.cc:1397] T 6f3968a617ee4c189de347e185fae427 P 2013510f40c14486a71a3f2d3668d06a: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260808 04:04:51.122212 16900 ts_tablet_manager.cc:1397] T 6f3968a617ee4c189de347e185fae427 P c230911ad282421cabd56a069e80976f: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260808 04:04:51.125931 16899 raft_consensus.cc:348] T 6f3968a617ee4c189de347e185fae427 P 442fa005e54743748ca0af7671c4a79a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2013510f40c14486a71a3f2d3668d06a" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 35837 } } peers { permanent_uuid: "442fa005e54743748ca0af7671c4a79a" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 39807 } } peers { permanent_uuid: "c230911ad282421cabd56a069e80976f" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 43655 } }
I20260808 04:04:51.126361 16899 raft_consensus.cc:374] T 6f3968a617ee4c189de347e185fae427 P 442fa005e54743748ca0af7671c4a79a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:04:51.126464 16899 raft_consensus.cc:729] T 6f3968a617ee4c189de347e185fae427 P 442fa005e54743748ca0af7671c4a79a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 442fa005e54743748ca0af7671c4a79a, State: Initialized, Role: FOLLOWER
I20260808 04:04:51.127072 16899 consensus_queue.cc:260] T 6f3968a617ee4c189de347e185fae427 P 442fa005e54743748ca0af7671c4a79a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2013510f40c14486a71a3f2d3668d06a" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 35837 } } peers { permanent_uuid: "442fa005e54743748ca0af7671c4a79a" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 39807 } } peers { permanent_uuid: "c230911ad282421cabd56a069e80976f" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 43655 } }
I20260808 04:04:51.128185 16648 heartbeater.cc:503] Master 127.5.83.62:33481 was elected leader, sending a full tablet report...
I20260808 04:04:51.129171 16902 raft_consensus.cc:348] T 6f3968a617ee4c189de347e185fae427 P 2013510f40c14486a71a3f2d3668d06a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2013510f40c14486a71a3f2d3668d06a" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 35837 } } peers { permanent_uuid: "442fa005e54743748ca0af7671c4a79a" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 39807 } } peers { permanent_uuid: "c230911ad282421cabd56a069e80976f" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 43655 } }
I20260808 04:04:51.129514 16902 raft_consensus.cc:374] T 6f3968a617ee4c189de347e185fae427 P 2013510f40c14486a71a3f2d3668d06a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:04:51.129609 16902 raft_consensus.cc:729] T 6f3968a617ee4c189de347e185fae427 P 2013510f40c14486a71a3f2d3668d06a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2013510f40c14486a71a3f2d3668d06a, State: Initialized, Role: FOLLOWER
I20260808 04:04:51.130177 16899 ts_tablet_manager.cc:1428] T 6f3968a617ee4c189de347e185fae427 P 442fa005e54743748ca0af7671c4a79a: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260808 04:04:51.130270 16902 consensus_queue.cc:260] T 6f3968a617ee4c189de347e185fae427 P 2013510f40c14486a71a3f2d3668d06a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2013510f40c14486a71a3f2d3668d06a" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 35837 } } peers { permanent_uuid: "442fa005e54743748ca0af7671c4a79a" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 39807 } } peers { permanent_uuid: "c230911ad282421cabd56a069e80976f" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 43655 } }
I20260808 04:04:51.131994 16902 ts_tablet_manager.cc:1428] T 6f3968a617ee4c189de347e185fae427 P 2013510f40c14486a71a3f2d3668d06a: Time spent starting tablet: real 0.010s user 0.011s sys 0.001s
I20260808 04:04:51.132118 16900 raft_consensus.cc:348] T 6f3968a617ee4c189de347e185fae427 P c230911ad282421cabd56a069e80976f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2013510f40c14486a71a3f2d3668d06a" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 35837 } } peers { permanent_uuid: "442fa005e54743748ca0af7671c4a79a" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 39807 } } peers { permanent_uuid: "c230911ad282421cabd56a069e80976f" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 43655 } }
I20260808 04:04:51.132513 16900 raft_consensus.cc:374] T 6f3968a617ee4c189de347e185fae427 P c230911ad282421cabd56a069e80976f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:04:51.132611 16900 raft_consensus.cc:729] T 6f3968a617ee4c189de347e185fae427 P c230911ad282421cabd56a069e80976f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c230911ad282421cabd56a069e80976f, State: Initialized, Role: FOLLOWER
I20260808 04:04:51.133245 16900 consensus_queue.cc:260] T 6f3968a617ee4c189de347e185fae427 P c230911ad282421cabd56a069e80976f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2013510f40c14486a71a3f2d3668d06a" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 35837 } } peers { permanent_uuid: "442fa005e54743748ca0af7671c4a79a" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 39807 } } peers { permanent_uuid: "c230911ad282421cabd56a069e80976f" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 43655 } }
I20260808 04:04:51.134596 16863 heartbeater.cc:503] Master 127.5.83.62:33481 was elected leader, sending a full tablet report...
I20260808 04:04:51.135232 16900 ts_tablet_manager.cc:1428] T 6f3968a617ee4c189de347e185fae427 P c230911ad282421cabd56a069e80976f: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260808 04:04:51.167320 16911 raft_consensus.cc:482] T 6f3968a617ee4c189de347e185fae427 P 442fa005e54743748ca0af7671c4a79a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:04:51.167618 16911 raft_consensus.cc:504] T 6f3968a617ee4c189de347e185fae427 P 442fa005e54743748ca0af7671c4a79a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2013510f40c14486a71a3f2d3668d06a" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 35837 } } peers { permanent_uuid: "442fa005e54743748ca0af7671c4a79a" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 39807 } } peers { permanent_uuid: "c230911ad282421cabd56a069e80976f" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 43655 } }
I20260808 04:04:51.169425 16911 leader_election.cc:288] T 6f3968a617ee4c189de347e185fae427 P 442fa005e54743748ca0af7671c4a79a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2013510f40c14486a71a3f2d3668d06a (127.5.83.1:35837), c230911ad282421cabd56a069e80976f (127.5.83.3:43655)
I20260808 04:04:51.176287 16898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:51.170020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.2:34817 (local address 127.5.83.3:43655)
0808 04:04:51.170217 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:51.170223 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:51.170848 (+ 625us) server_negotiation.cc:407] Connection header received
0808 04:04:51.170996 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:51.171000 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:51.171063 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:51.171151 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:04:51.171804 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:51.172500 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:51.174552 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:51.175289 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:51.175443 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:51.175545 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:51.175999 (+ 454us) server_negotiation.cc:299] Negotiation successful
0808 04:04:51.176101 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 04:04:51.176704 16915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:51.169824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.3:43655 (local address 127.5.83.2:34817)
0808 04:04:51.170729 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:51.170748 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:51.170877 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:51.171191 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:51.171200 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:51.171224 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:04:51.171646 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:51.171656 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:51.172630 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:51.172636 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:51.174321 (+ 1685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:51.174339 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:51.175571 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:51.175576 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:51.175866 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:51.175892 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 04:04:51.175969 (+ 77us) client_negotiation.cc:240] Negotiation successful
0808 04:04:51.176177 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":792,"mutex_wait_us":118,"thread_start_us":613,"threads_started":1}
I20260808 04:04:51.176728 16914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:51.169750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:35837 (local address 127.5.83.2:44693)
0808 04:04:51.170596 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:51.170615 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:51.170759 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:51.171107 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:51.171112 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:51.171137 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:04:51.171607 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:51.171617 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:51.172616 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:51.172621 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:51.174290 (+ 1669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:51.174305 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:51.175521 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:51.175530 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:51.175795 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:51.175867 (+ 72us) client_negotiation.cc:769] Sending connection context
0808 04:04:51.175969 (+ 102us) client_negotiation.cc:240] Negotiation successful
0808 04:04:51.176177 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":689,"mutex_wait_us":150,"spinlock_wait_cycles":20736,"thread_start_us":415,"threads_started":1}
I20260808 04:04:51.176863 16892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:51.169711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.2:44693 (local address 127.5.83.1:35837)
0808 04:04:51.169925 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:51.169932 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:51.170726 (+ 794us) server_negotiation.cc:407] Connection header received
0808 04:04:51.170908 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:51.170914 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:51.171007 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:51.171072 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:04:51.171776 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:51.172492 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:51.174517 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:51.175268 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:51.175396 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:51.175500 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:51.176496 (+ 996us) server_negotiation.cc:299] Negotiation successful
0808 04:04:51.176619 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 04:04:51.177680 16355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f3968a617ee4c189de347e185fae427" candidate_uuid: "442fa005e54743748ca0af7671c4a79a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2013510f40c14486a71a3f2d3668d06a" is_pre_election: true
I20260808 04:04:51.177659 16781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f3968a617ee4c189de347e185fae427" candidate_uuid: "442fa005e54743748ca0af7671c4a79a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c230911ad282421cabd56a069e80976f" is_pre_election: true
I20260808 04:04:51.178129 16355 raft_consensus.cc:2454] T 6f3968a617ee4c189de347e185fae427 P 2013510f40c14486a71a3f2d3668d06a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 442fa005e54743748ca0af7671c4a79a in term 0.
I20260808 04:04:51.178128 16781 raft_consensus.cc:2454] T 6f3968a617ee4c189de347e185fae427 P c230911ad282421cabd56a069e80976f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 442fa005e54743748ca0af7671c4a79a in term 0.
I20260808 04:04:51.178676 16470 leader_election.cc:302] T 6f3968a617ee4c189de347e185fae427 P 442fa005e54743748ca0af7671c4a79a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2013510f40c14486a71a3f2d3668d06a, 442fa005e54743748ca0af7671c4a79a; no voters:
I20260808 04:04:51.179051 16911 raft_consensus.cc:2781] T 6f3968a617ee4c189de347e185fae427 P 442fa005e54743748ca0af7671c4a79a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:04:51.179221 16911 raft_consensus.cc:482] T 6f3968a617ee4c189de347e185fae427 P 442fa005e54743748ca0af7671c4a79a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:04:51.179324 16911 raft_consensus.cc:3037] T 6f3968a617ee4c189de347e185fae427 P 442fa005e54743748ca0af7671c4a79a [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:04:51.181720 16911 raft_consensus.cc:504] T 6f3968a617ee4c189de347e185fae427 P 442fa005e54743748ca0af7671c4a79a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2013510f40c14486a71a3f2d3668d06a" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 35837 } } peers { permanent_uuid: "442fa005e54743748ca0af7671c4a79a" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 39807 } } peers { permanent_uuid: "c230911ad282421cabd56a069e80976f" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 43655 } }
I20260808 04:04:51.182297 16911 leader_election.cc:288] T 6f3968a617ee4c189de347e185fae427 P 442fa005e54743748ca0af7671c4a79a [CANDIDATE]: Term 1 election: Requested vote from peers 2013510f40c14486a71a3f2d3668d06a (127.5.83.1:35837), c230911ad282421cabd56a069e80976f (127.5.83.3:43655)
I20260808 04:04:51.182647 16355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f3968a617ee4c189de347e185fae427" candidate_uuid: "442fa005e54743748ca0af7671c4a79a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2013510f40c14486a71a3f2d3668d06a"
I20260808 04:04:51.182760 16781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f3968a617ee4c189de347e185fae427" candidate_uuid: "442fa005e54743748ca0af7671c4a79a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c230911ad282421cabd56a069e80976f"
I20260808 04:04:51.182858 16355 raft_consensus.cc:3037] T 6f3968a617ee4c189de347e185fae427 P 2013510f40c14486a71a3f2d3668d06a [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:04:51.182955 16781 raft_consensus.cc:3037] T 6f3968a617ee4c189de347e185fae427 P c230911ad282421cabd56a069e80976f [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:04:51.186275 16355 raft_consensus.cc:2454] T 6f3968a617ee4c189de347e185fae427 P 2013510f40c14486a71a3f2d3668d06a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 442fa005e54743748ca0af7671c4a79a in term 1.
I20260808 04:04:51.186275 16781 raft_consensus.cc:2454] T 6f3968a617ee4c189de347e185fae427 P c230911ad282421cabd56a069e80976f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 442fa005e54743748ca0af7671c4a79a in term 1.
I20260808 04:04:51.186666 16470 leader_election.cc:302] T 6f3968a617ee4c189de347e185fae427 P 442fa005e54743748ca0af7671c4a79a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2013510f40c14486a71a3f2d3668d06a, 442fa005e54743748ca0af7671c4a79a; no voters:
I20260808 04:04:51.186972 16911 raft_consensus.cc:2781] T 6f3968a617ee4c189de347e185fae427 P 442fa005e54743748ca0af7671c4a79a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:04:51.187409 16911 raft_consensus.cc:686] T 6f3968a617ee4c189de347e185fae427 P 442fa005e54743748ca0af7671c4a79a [term 1 LEADER]: Becoming Leader. State: Replica: 442fa005e54743748ca0af7671c4a79a, State: Running, Role: LEADER
I20260808 04:04:51.187892 16911 consensus_queue.cc:237] T 6f3968a617ee4c189de347e185fae427 P 442fa005e54743748ca0af7671c4a79a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2013510f40c14486a71a3f2d3668d06a" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 35837 } } peers { permanent_uuid: "442fa005e54743748ca0af7671c4a79a" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 39807 } } peers { permanent_uuid: "c230911ad282421cabd56a069e80976f" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 43655 } }
I20260808 04:04:51.192523 16108 catalog_manager.cc:5495] T 6f3968a617ee4c189de347e185fae427 P 442fa005e54743748ca0af7671c4a79a reported cstate change: term changed from 0 to 1, leader changed from <none> to 442fa005e54743748ca0af7671c4a79a (127.5.83.2). New cstate: current_term: 1 leader_uuid: "442fa005e54743748ca0af7671c4a79a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2013510f40c14486a71a3f2d3668d06a" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 35837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "442fa005e54743748ca0af7671c4a79a" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 39807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c230911ad282421cabd56a069e80976f" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 43655 } health_report { overall_health: UNKNOWN } } }
W20260808 04:04:51.211351 16439 tablet.cc:2369] T 6f3968a617ee4c189de347e185fae427 P 2013510f40c14486a71a3f2d3668d06a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 04:04:51.213706 16864 tablet.cc:2369] T 6f3968a617ee4c189de347e185fae427 P c230911ad282421cabd56a069e80976f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 04:04:51.216830 16649 tablet.cc:2369] T 6f3968a617ee4c189de347e185fae427 P 442fa005e54743748ca0af7671c4a79a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:04:51.217072 16890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:51.211561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34684 (local address 127.5.83.2:39807)
0808 04:04:51.211814 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:51.211820 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:51.211843 (+ 23us) server_negotiation.cc:407] Connection header received
0808 04:04:51.212056 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:51.212062 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:51.212172 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:51.212310 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:04:51.212595 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:51.213307 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:51.214954 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:51.215355 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:51.215454 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:51.215869 (+ 415us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:04:51.216370 (+ 501us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:04:51.216493 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:51.216644 (+ 151us) server_negotiation.cc:299] Negotiation successful
0808 04:04:51.216726 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260808 04:04:51.224563 16781 raft_consensus.cc:1261] T 6f3968a617ee4c189de347e185fae427 P c230911ad282421cabd56a069e80976f [term 1 FOLLOWER]: Refusing update from remote peer 442fa005e54743748ca0af7671c4a79a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:04:51.224611 16355 raft_consensus.cc:1261] T 6f3968a617ee4c189de347e185fae427 P 2013510f40c14486a71a3f2d3668d06a [term 1 FOLLOWER]: Refusing update from remote peer 442fa005e54743748ca0af7671c4a79a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:04:51.225456 16911 consensus_queue.cc:1035] T 6f3968a617ee4c189de347e185fae427 P 442fa005e54743748ca0af7671c4a79a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2013510f40c14486a71a3f2d3668d06a" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 35837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:04:51.225737 16916 consensus_queue.cc:1035] T 6f3968a617ee4c189de347e185fae427 P 442fa005e54743748ca0af7671c4a79a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c230911ad282421cabd56a069e80976f" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 43655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:04:51.237258 16920 mvcc.cc:205] Tried to move back new op lower bound from 7316119106447286272 to 7316119106309427200. Current Snapshot: MvccSnapshot[applied={T|T < 7316119106447286272}]
I20260808 04:04:51.241498 16924 mvcc.cc:205] Tried to move back new op lower bound from 7316119106447286272 to 7316119106309427200. Current Snapshot: MvccSnapshot[applied={T|T < 7316119106447286272}]
I20260808 04:04:51.256780 16112 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:04:51.257061 16112 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:04:51.260854 16112 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9f9dc08ed53644aa9bd23b443d2ae405: Sending DeleteTablet for 3 replicas of tablet 6f3968a617ee4c189de347e185fae427
I20260808 04:04:51.261834 16550 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f3968a617ee4c189de347e185fae427 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:04:51 UTC) from {username='oryx', principal='oryx/127.5.83.62@KRBTEST.COM'} at 127.0.0.1:34674
I20260808 04:04:51.261809 16313 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f3968a617ee4c189de347e185fae427 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:04:51 UTC) from {username='oryx', principal='oryx/127.5.83.62@KRBTEST.COM'} at 127.0.0.1:33308
I20260808 04:04:51.262094 16752 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f3968a617ee4c189de347e185fae427 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:04:51 UTC) from {username='oryx', principal='oryx/127.5.83.62@KRBTEST.COM'} at 127.0.0.1:41412
I20260808 04:04:51.263795 16112 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55474:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:04:51.264660 16112 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:04:51.272442 16547 tablet_service.cc:1467] Processing CreateTablet for tablet 543727dd35c145599e002e02592ba63e (DEFAULT_TABLE table=test-table [id=564ec0e52f7d4afda17f6e31e085683e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:04:51.273033 16547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 543727dd35c145599e002e02592ba63e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:04:51.275697 16312 tablet_service.cc:1467] Processing CreateTablet for tablet 543727dd35c145599e002e02592ba63e (DEFAULT_TABLE table=test-table [id=564ec0e52f7d4afda17f6e31e085683e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:04:51.275907 16751 tablet_service.cc:1467] Processing CreateTablet for tablet 543727dd35c145599e002e02592ba63e (DEFAULT_TABLE table=test-table [id=564ec0e52f7d4afda17f6e31e085683e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:04:51.276350 16312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 543727dd35c145599e002e02592ba63e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:04:51.276533 16751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 543727dd35c145599e002e02592ba63e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:04:51.283892 16902 tablet_bootstrap.cc:492] T 543727dd35c145599e002e02592ba63e P 2013510f40c14486a71a3f2d3668d06a: Bootstrap starting.
I20260808 04:04:51.285552 16902 tablet_bootstrap.cc:654] T 543727dd35c145599e002e02592ba63e P 2013510f40c14486a71a3f2d3668d06a: Neither blocks nor log segments found. Creating new log.
I20260808 04:04:51.290073 16900 tablet_bootstrap.cc:492] T 543727dd35c145599e002e02592ba63e P c230911ad282421cabd56a069e80976f: Bootstrap starting.
I20260808 04:04:51.291940 16900 tablet_bootstrap.cc:654] T 543727dd35c145599e002e02592ba63e P c230911ad282421cabd56a069e80976f: Neither blocks nor log segments found. Creating new log.
I20260808 04:04:51.292910 16899 tablet_bootstrap.cc:492] T 543727dd35c145599e002e02592ba63e P 442fa005e54743748ca0af7671c4a79a: Bootstrap starting.
I20260808 04:04:51.294610 16899 tablet_bootstrap.cc:654] T 543727dd35c145599e002e02592ba63e P 442fa005e54743748ca0af7671c4a79a: Neither blocks nor log segments found. Creating new log.
I20260808 04:04:51.296396 16935 tablet_replica.cc:330] T 6f3968a617ee4c189de347e185fae427 P 442fa005e54743748ca0af7671c4a79a: stopping tablet replica
I20260808 04:04:51.296883 16935 raft_consensus.cc:2229] T 6f3968a617ee4c189de347e185fae427 P 442fa005e54743748ca0af7671c4a79a [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:04:51.297248 16899 tablet_bootstrap.cc:492] T 543727dd35c145599e002e02592ba63e P 442fa005e54743748ca0af7671c4a79a: No bootstrap required, opened a new log
I20260808 04:04:51.297410 16899 ts_tablet_manager.cc:1397] T 543727dd35c145599e002e02592ba63e P 442fa005e54743748ca0af7671c4a79a: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260808 04:04:51.297580 16935 raft_consensus.cc:2258] T 6f3968a617ee4c189de347e185fae427 P 442fa005e54743748ca0af7671c4a79a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:04:51.298652 16899 raft_consensus.cc:348] T 543727dd35c145599e002e02592ba63e P 442fa005e54743748ca0af7671c4a79a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2013510f40c14486a71a3f2d3668d06a" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 35837 } } peers { permanent_uuid: "442fa005e54743748ca0af7671c4a79a" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 39807 } } peers { permanent_uuid: "c230911ad282421cabd56a069e80976f" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 43655 } }
I20260808 04:04:51.298903 16899 raft_consensus.cc:374] T 543727dd35c145599e002e02592ba63e P 442fa005e54743748ca0af7671c4a79a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:04:51.299005 16899 raft_consensus.cc:729] T 543727dd35c145599e002e02592ba63e P 442fa005e54743748ca0af7671c4a79a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 442fa005e54743748ca0af7671c4a79a, State: Initialized, Role: FOLLOWER
I20260808 04:04:51.299283 16899 consensus_queue.cc:260] T 543727dd35c145599e002e02592ba63e P 442fa005e54743748ca0af7671c4a79a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2013510f40c14486a71a3f2d3668d06a" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 35837 } } peers { permanent_uuid: "442fa005e54743748ca0af7671c4a79a" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 39807 } } peers { permanent_uuid: "c230911ad282421cabd56a069e80976f" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 43655 } }
I20260808 04:04:51.299851 16899 ts_tablet_manager.cc:1428] T 543727dd35c145599e002e02592ba63e P 442fa005e54743748ca0af7671c4a79a: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260808 04:04:51.300935 16900 tablet_bootstrap.cc:492] T 543727dd35c145599e002e02592ba63e P c230911ad282421cabd56a069e80976f: No bootstrap required, opened a new log
I20260808 04:04:51.301095 16900 ts_tablet_manager.cc:1397] T 543727dd35c145599e002e02592ba63e P c230911ad282421cabd56a069e80976f: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.003s
I20260808 04:04:51.301924 16900 raft_consensus.cc:348] T 543727dd35c145599e002e02592ba63e P c230911ad282421cabd56a069e80976f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2013510f40c14486a71a3f2d3668d06a" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 35837 } } peers { permanent_uuid: "442fa005e54743748ca0af7671c4a79a" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 39807 } } peers { permanent_uuid: "c230911ad282421cabd56a069e80976f" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 43655 } }
I20260808 04:04:51.302177 16900 raft_consensus.cc:374] T 543727dd35c145599e002e02592ba63e P c230911ad282421cabd56a069e80976f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:04:51.302268 16900 raft_consensus.cc:729] T 543727dd35c145599e002e02592ba63e P c230911ad282421cabd56a069e80976f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c230911ad282421cabd56a069e80976f, State: Initialized, Role: FOLLOWER
I20260808 04:04:51.302523 16900 consensus_queue.cc:260] T 543727dd35c145599e002e02592ba63e P c230911ad282421cabd56a069e80976f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2013510f40c14486a71a3f2d3668d06a" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 35837 } } peers { permanent_uuid: "442fa005e54743748ca0af7671c4a79a" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 39807 } } peers { permanent_uuid: "c230911ad282421cabd56a069e80976f" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 43655 } }
I20260808 04:04:51.303081 16900 ts_tablet_manager.cc:1428] T 543727dd35c145599e002e02592ba63e P c230911ad282421cabd56a069e80976f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260808 04:04:51.304765 16933 tablet_replica.cc:330] T 6f3968a617ee4c189de347e185fae427 P 2013510f40c14486a71a3f2d3668d06a: stopping tablet replica
I20260808 04:04:51.305231 16933 raft_consensus.cc:2229] T 6f3968a617ee4c189de347e185fae427 P 2013510f40c14486a71a3f2d3668d06a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:04:51.305738 16933 raft_consensus.cc:2258] T 6f3968a617ee4c189de347e185fae427 P 2013510f40c14486a71a3f2d3668d06a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:04:51.306788 16902 tablet_bootstrap.cc:492] T 543727dd35c145599e002e02592ba63e P 2013510f40c14486a71a3f2d3668d06a: No bootstrap required, opened a new log
I20260808 04:04:51.306936 16902 ts_tablet_manager.cc:1397] T 543727dd35c145599e002e02592ba63e P 2013510f40c14486a71a3f2d3668d06a: Time spent bootstrapping tablet: real 0.023s user 0.000s sys 0.003s
I20260808 04:04:51.307741 16902 raft_consensus.cc:348] T 543727dd35c145599e002e02592ba63e P 2013510f40c14486a71a3f2d3668d06a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2013510f40c14486a71a3f2d3668d06a" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 35837 } } peers { permanent_uuid: "442fa005e54743748ca0af7671c4a79a" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 39807 } } peers { permanent_uuid: "c230911ad282421cabd56a069e80976f" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 43655 } }
I20260808 04:04:51.307981 16902 raft_consensus.cc:374] T 543727dd35c145599e002e02592ba63e P 2013510f40c14486a71a3f2d3668d06a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:04:51.308063 16902 raft_consensus.cc:729] T 543727dd35c145599e002e02592ba63e P 2013510f40c14486a71a3f2d3668d06a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2013510f40c14486a71a3f2d3668d06a, State: Initialized, Role: FOLLOWER
I20260808 04:04:51.308768 16902 consensus_queue.cc:260] T 543727dd35c145599e002e02592ba63e P 2013510f40c14486a71a3f2d3668d06a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2013510f40c14486a71a3f2d3668d06a" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 35837 } } peers { permanent_uuid: "442fa005e54743748ca0af7671c4a79a" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 39807 } } peers { permanent_uuid: "c230911ad282421cabd56a069e80976f" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 43655 } }
I20260808 04:04:51.309442 16902 ts_tablet_manager.cc:1428] T 543727dd35c145599e002e02592ba63e P 2013510f40c14486a71a3f2d3668d06a: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260808 04:04:51.311182 16935 ts_tablet_manager.cc:1905] T 6f3968a617ee4c189de347e185fae427 P 442fa005e54743748ca0af7671c4a79a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:04:51.312942 16933 ts_tablet_manager.cc:1905] T 6f3968a617ee4c189de347e185fae427 P 2013510f40c14486a71a3f2d3668d06a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:04:51.320310 16933 ts_tablet_manager.cc:1918] T 6f3968a617ee4c189de347e185fae427 P 2013510f40c14486a71a3f2d3668d06a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:04:51.320484 16933 log.cc:1196] T 6f3968a617ee4c189de347e185fae427 P 2013510f40c14486a71a3f2d3668d06a: Deleting WAL directory at /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/minicluster-data/ts-0/wal/wals/6f3968a617ee4c189de347e185fae427
I20260808 04:04:51.321033 16933 ts_tablet_manager.cc:1939] T 6f3968a617ee4c189de347e185fae427 P 2013510f40c14486a71a3f2d3668d06a: Deleting consensus metadata
I20260808 04:04:51.321650 16936 tablet_replica.cc:330] T 6f3968a617ee4c189de347e185fae427 P c230911ad282421cabd56a069e80976f: stopping tablet replica
I20260808 04:04:51.322116 16936 raft_consensus.cc:2229] T 6f3968a617ee4c189de347e185fae427 P c230911ad282421cabd56a069e80976f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:04:51.322620 16936 raft_consensus.cc:2258] T 6f3968a617ee4c189de347e185fae427 P c230911ad282421cabd56a069e80976f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:04:51.322625 16086 catalog_manager.cc:4841] TS 2013510f40c14486a71a3f2d3668d06a (127.5.83.1:35837): tablet 6f3968a617ee4c189de347e185fae427 (table test-table [id=83d5eecc394644fea0b1462545912a87]) successfully deleted
I20260808 04:04:51.324604 16935 ts_tablet_manager.cc:1918] T 6f3968a617ee4c189de347e185fae427 P 442fa005e54743748ca0af7671c4a79a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:04:51.324755 16935 log.cc:1196] T 6f3968a617ee4c189de347e185fae427 P 442fa005e54743748ca0af7671c4a79a: Deleting WAL directory at /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/minicluster-data/ts-1/wal/wals/6f3968a617ee4c189de347e185fae427
I20260808 04:04:51.325240 16935 ts_tablet_manager.cc:1939] T 6f3968a617ee4c189de347e185fae427 P 442fa005e54743748ca0af7671c4a79a: Deleting consensus metadata
I20260808 04:04:51.325296 16936 ts_tablet_manager.cc:1905] T 6f3968a617ee4c189de347e185fae427 P c230911ad282421cabd56a069e80976f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:04:51.326522 16087 catalog_manager.cc:4841] TS 442fa005e54743748ca0af7671c4a79a (127.5.83.2:39807): tablet 6f3968a617ee4c189de347e185fae427 (table test-table [id=83d5eecc394644fea0b1462545912a87]) successfully deleted
I20260808 04:04:51.332089 16936 ts_tablet_manager.cc:1918] T 6f3968a617ee4c189de347e185fae427 P c230911ad282421cabd56a069e80976f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:04:51.332249 16936 log.cc:1196] T 6f3968a617ee4c189de347e185fae427 P c230911ad282421cabd56a069e80976f: Deleting WAL directory at /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/minicluster-data/ts-2/wal/wals/6f3968a617ee4c189de347e185fae427
I20260808 04:04:51.332787 16936 ts_tablet_manager.cc:1939] T 6f3968a617ee4c189de347e185fae427 P c230911ad282421cabd56a069e80976f: Deleting consensus metadata
I20260808 04:04:51.334489 16088 catalog_manager.cc:4841] TS c230911ad282421cabd56a069e80976f (127.5.83.3:43655): tablet 6f3968a617ee4c189de347e185fae427 (table test-table [id=83d5eecc394644fea0b1462545912a87]) successfully deleted
I20260808 04:04:51.368569 16930 raft_consensus.cc:482] T 543727dd35c145599e002e02592ba63e P 442fa005e54743748ca0af7671c4a79a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:04:51.368808 16930 raft_consensus.cc:504] T 543727dd35c145599e002e02592ba63e P 442fa005e54743748ca0af7671c4a79a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2013510f40c14486a71a3f2d3668d06a" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 35837 } } peers { permanent_uuid: "442fa005e54743748ca0af7671c4a79a" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 39807 } } peers { permanent_uuid: "c230911ad282421cabd56a069e80976f" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 43655 } }
I20260808 04:04:51.369686 16930 leader_election.cc:288] T 543727dd35c145599e002e02592ba63e P 442fa005e54743748ca0af7671c4a79a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2013510f40c14486a71a3f2d3668d06a (127.5.83.1:35837), c230911ad282421cabd56a069e80976f (127.5.83.3:43655)
I20260808 04:04:51.370223 16781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "543727dd35c145599e002e02592ba63e" candidate_uuid: "442fa005e54743748ca0af7671c4a79a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c230911ad282421cabd56a069e80976f" is_pre_election: true
I20260808 04:04:51.370260 16355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "543727dd35c145599e002e02592ba63e" candidate_uuid: "442fa005e54743748ca0af7671c4a79a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2013510f40c14486a71a3f2d3668d06a" is_pre_election: true
I20260808 04:04:51.370473 16781 raft_consensus.cc:2454] T 543727dd35c145599e002e02592ba63e P c230911ad282421cabd56a069e80976f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 442fa005e54743748ca0af7671c4a79a in term 0.
I20260808 04:04:51.370534 16355 raft_consensus.cc:2454] T 543727dd35c145599e002e02592ba63e P 2013510f40c14486a71a3f2d3668d06a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 442fa005e54743748ca0af7671c4a79a in term 0.
I20260808 04:04:51.370910 16473 leader_election.cc:302] T 543727dd35c145599e002e02592ba63e P 442fa005e54743748ca0af7671c4a79a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 442fa005e54743748ca0af7671c4a79a, c230911ad282421cabd56a069e80976f; no voters:
I20260808 04:04:51.371225 16930 raft_consensus.cc:2781] T 543727dd35c145599e002e02592ba63e P 442fa005e54743748ca0af7671c4a79a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:04:51.371364 16930 raft_consensus.cc:482] T 543727dd35c145599e002e02592ba63e P 442fa005e54743748ca0af7671c4a79a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:04:51.371466 16930 raft_consensus.cc:3037] T 543727dd35c145599e002e02592ba63e P 442fa005e54743748ca0af7671c4a79a [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:04:51.373756 16930 raft_consensus.cc:504] T 543727dd35c145599e002e02592ba63e P 442fa005e54743748ca0af7671c4a79a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2013510f40c14486a71a3f2d3668d06a" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 35837 } } peers { permanent_uuid: "442fa005e54743748ca0af7671c4a79a" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 39807 } } peers { permanent_uuid: "c230911ad282421cabd56a069e80976f" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 43655 } }
I20260808 04:04:51.374359 16930 leader_election.cc:288] T 543727dd35c145599e002e02592ba63e P 442fa005e54743748ca0af7671c4a79a [CANDIDATE]: Term 1 election: Requested vote from peers 2013510f40c14486a71a3f2d3668d06a (127.5.83.1:35837), c230911ad282421cabd56a069e80976f (127.5.83.3:43655)
I20260808 04:04:51.374864 16781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "543727dd35c145599e002e02592ba63e" candidate_uuid: "442fa005e54743748ca0af7671c4a79a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c230911ad282421cabd56a069e80976f"
I20260808 04:04:51.374900 16355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "543727dd35c145599e002e02592ba63e" candidate_uuid: "442fa005e54743748ca0af7671c4a79a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2013510f40c14486a71a3f2d3668d06a"
I20260808 04:04:51.375044 16781 raft_consensus.cc:3037] T 543727dd35c145599e002e02592ba63e P c230911ad282421cabd56a069e80976f [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:04:51.375093 16355 raft_consensus.cc:3037] T 543727dd35c145599e002e02592ba63e P 2013510f40c14486a71a3f2d3668d06a [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:04:51.377154 16781 raft_consensus.cc:2454] T 543727dd35c145599e002e02592ba63e P c230911ad282421cabd56a069e80976f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 442fa005e54743748ca0af7671c4a79a in term 1.
I20260808 04:04:51.377558 16473 leader_election.cc:302] T 543727dd35c145599e002e02592ba63e P 442fa005e54743748ca0af7671c4a79a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 442fa005e54743748ca0af7671c4a79a, c230911ad282421cabd56a069e80976f; no voters:
I20260808 04:04:51.377846 16930 raft_consensus.cc:2781] T 543727dd35c145599e002e02592ba63e P 442fa005e54743748ca0af7671c4a79a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:04:51.378014 16930 raft_consensus.cc:686] T 543727dd35c145599e002e02592ba63e P 442fa005e54743748ca0af7671c4a79a [term 1 LEADER]: Becoming Leader. State: Replica: 442fa005e54743748ca0af7671c4a79a, State: Running, Role: LEADER
I20260808 04:04:51.378082 16355 raft_consensus.cc:2454] T 543727dd35c145599e002e02592ba63e P 2013510f40c14486a71a3f2d3668d06a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 442fa005e54743748ca0af7671c4a79a in term 1.
I20260808 04:04:51.378325 16930 consensus_queue.cc:237] T 543727dd35c145599e002e02592ba63e P 442fa005e54743748ca0af7671c4a79a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2013510f40c14486a71a3f2d3668d06a" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 35837 } } peers { permanent_uuid: "442fa005e54743748ca0af7671c4a79a" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 39807 } } peers { permanent_uuid: "c230911ad282421cabd56a069e80976f" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 43655 } }
I20260808 04:04:51.381045 16108 catalog_manager.cc:5495] T 543727dd35c145599e002e02592ba63e P 442fa005e54743748ca0af7671c4a79a reported cstate change: term changed from 0 to 1, leader changed from <none> to 442fa005e54743748ca0af7671c4a79a (127.5.83.2). New cstate: current_term: 1 leader_uuid: "442fa005e54743748ca0af7671c4a79a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2013510f40c14486a71a3f2d3668d06a" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 35837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "442fa005e54743748ca0af7671c4a79a" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 39807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c230911ad282421cabd56a069e80976f" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 43655 } health_report { overall_health: UNKNOWN } } }
Aug 08 04:04:51 dist-test-slave-1l3l krb5kdc[16047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161889, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.83.62@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM
I20260808 04:04:51.451421 16254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:51.435169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55480 (local address 127.5.83.62:33481)
0808 04:04:51.435389 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:51.435399 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:51.436331 (+ 932us) server_negotiation.cc:407] Connection header received
0808 04:04:51.436520 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:51.436525 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:51.436583 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:51.436687 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:51.437999 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:51.438764 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:51.440031 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:51.440426 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:51.440529 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:51.446342 (+ 5813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:51.446349 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:51.446352 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:51.446381 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:51.448874 (+ 2493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:51.449522 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:51.449526 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:51.449528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:51.449642 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:51.449990 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:51.449995 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:51.449998 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:51.450239 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:51.450364 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:51.450791 (+ 427us) server_negotiation.cc:299] Negotiation successful
0808 04:04:51.451061 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":80,"server-negotiator.queue_time_us":97}
I20260808 04:04:51.451439 16950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:51.435470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:33481 (local address 127.0.0.1:55480)
0808 04:04:51.436233 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:51.436249 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:51.436404 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:51.436874 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:51.436878 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:51.437611 (+ 733us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:04:51.437858 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:51.437868 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:51.438909 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:51.438914 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:51.439907 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:51.439920 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:51.440585 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:51.440591 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:51.440735 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:51.444040 (+ 3305us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:04:51.444059 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:04:51.446165 (+ 2106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:04:51.448997 (+ 2832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:51.449009 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:51.449017 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:51.449407 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:51.449740 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:51.449745 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:51.449749 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:51.449879 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:51.450357 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:04:51.450365 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:04:51.450524 (+ 159us) client_negotiation.cc:769] Sending connection context
0808 04:04:51.450660 (+ 136us) client_negotiation.cc:240] Negotiation successful
0808 04:04:51.450935 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":434,"threads_started":1}
I20260808 04:04:51.457265 16108 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.5.83.62@KRBTEST.COM'} at 127.0.0.1:55480:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260808 04:04:51.458528 16108 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:04:51.465905 16752 tablet_service.cc:1467] Processing CreateTablet for tablet 3085c84faa37435c9ec121620dd037f4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=23db86c2e6844ef8b763d2aa5449f6c3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 04:04:51.466480 16752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3085c84faa37435c9ec121620dd037f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:04:51.466320 16313 tablet_service.cc:1467] Processing CreateTablet for tablet 3085c84faa37435c9ec121620dd037f4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=23db86c2e6844ef8b763d2aa5449f6c3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 04:04:51.466846 16313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3085c84faa37435c9ec121620dd037f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:04:51.468185 16550 tablet_service.cc:1467] Processing CreateTablet for tablet 3085c84faa37435c9ec121620dd037f4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=23db86c2e6844ef8b763d2aa5449f6c3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 04:04:51.468758 16550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3085c84faa37435c9ec121620dd037f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:04:51.473572 16900 tablet_bootstrap.cc:492] T 3085c84faa37435c9ec121620dd037f4 P c230911ad282421cabd56a069e80976f: Bootstrap starting.
I20260808 04:04:51.475340 16900 tablet_bootstrap.cc:654] T 3085c84faa37435c9ec121620dd037f4 P c230911ad282421cabd56a069e80976f: Neither blocks nor log segments found. Creating new log.
I20260808 04:04:51.481400 16902 tablet_bootstrap.cc:492] T 3085c84faa37435c9ec121620dd037f4 P 2013510f40c14486a71a3f2d3668d06a: Bootstrap starting.
I20260808 04:04:51.483093 16902 tablet_bootstrap.cc:654] T 3085c84faa37435c9ec121620dd037f4 P 2013510f40c14486a71a3f2d3668d06a: Neither blocks nor log segments found. Creating new log.
I20260808 04:04:51.486459 16899 tablet_bootstrap.cc:492] T 3085c84faa37435c9ec121620dd037f4 P 442fa005e54743748ca0af7671c4a79a: Bootstrap starting.
I20260808 04:04:51.488204 16899 tablet_bootstrap.cc:654] T 3085c84faa37435c9ec121620dd037f4 P 442fa005e54743748ca0af7671c4a79a: Neither blocks nor log segments found. Creating new log.
I20260808 04:04:51.495162 16899 tablet_bootstrap.cc:492] T 3085c84faa37435c9ec121620dd037f4 P 442fa005e54743748ca0af7671c4a79a: No bootstrap required, opened a new log
I20260808 04:04:51.495441 16899 ts_tablet_manager.cc:1397] T 3085c84faa37435c9ec121620dd037f4 P 442fa005e54743748ca0af7671c4a79a: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260808 04:04:51.495776 16902 tablet_bootstrap.cc:492] T 3085c84faa37435c9ec121620dd037f4 P 2013510f40c14486a71a3f2d3668d06a: No bootstrap required, opened a new log
I20260808 04:04:51.495935 16902 ts_tablet_manager.cc:1397] T 3085c84faa37435c9ec121620dd037f4 P 2013510f40c14486a71a3f2d3668d06a: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.004s
I20260808 04:04:51.496488 16899 raft_consensus.cc:348] T 3085c84faa37435c9ec121620dd037f4 P 442fa005e54743748ca0af7671c4a79a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2013510f40c14486a71a3f2d3668d06a" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 35837 } } peers { permanent_uuid: "442fa005e54743748ca0af7671c4a79a" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 39807 } } peers { permanent_uuid: "c230911ad282421cabd56a069e80976f" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 43655 } }
I20260808 04:04:51.496592 16902 raft_consensus.cc:348] T 3085c84faa37435c9ec121620dd037f4 P 2013510f40c14486a71a3f2d3668d06a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2013510f40c14486a71a3f2d3668d06a" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 35837 } } peers { permanent_uuid: "442fa005e54743748ca0af7671c4a79a" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 39807 } } peers { permanent_uuid: "c230911ad282421cabd56a069e80976f" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 43655 } }
I20260808 04:04:51.496824 16902 raft_consensus.cc:374] T 3085c84faa37435c9ec121620dd037f4 P 2013510f40c14486a71a3f2d3668d06a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:04:51.496896 16902 raft_consensus.cc:729] T 3085c84faa37435c9ec121620dd037f4 P 2013510f40c14486a71a3f2d3668d06a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2013510f40c14486a71a3f2d3668d06a, State: Initialized, Role: FOLLOWER
I20260808 04:04:51.496861 16899 raft_consensus.cc:374] T 3085c84faa37435c9ec121620dd037f4 P 442fa005e54743748ca0af7671c4a79a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:04:51.497109 16899 raft_consensus.cc:729] T 3085c84faa37435c9ec121620dd037f4 P 442fa005e54743748ca0af7671c4a79a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 442fa005e54743748ca0af7671c4a79a, State: Initialized, Role: FOLLOWER
I20260808 04:04:51.497155 16902 consensus_queue.cc:260] T 3085c84faa37435c9ec121620dd037f4 P 2013510f40c14486a71a3f2d3668d06a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2013510f40c14486a71a3f2d3668d06a" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 35837 } } peers { permanent_uuid: "442fa005e54743748ca0af7671c4a79a" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 39807 } } peers { permanent_uuid: "c230911ad282421cabd56a069e80976f" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 43655 } }
I20260808 04:04:51.497532 16900 tablet_bootstrap.cc:492] T 3085c84faa37435c9ec121620dd037f4 P c230911ad282421cabd56a069e80976f: No bootstrap required, opened a new log
I20260808 04:04:51.497699 16900 ts_tablet_manager.cc:1397] T 3085c84faa37435c9ec121620dd037f4 P c230911ad282421cabd56a069e80976f: Time spent bootstrapping tablet: real 0.024s user 0.000s sys 0.008s
I20260808 04:04:51.497514 16899 consensus_queue.cc:260] T 3085c84faa37435c9ec121620dd037f4 P 442fa005e54743748ca0af7671c4a79a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2013510f40c14486a71a3f2d3668d06a" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 35837 } } peers { permanent_uuid: "442fa005e54743748ca0af7671c4a79a" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 39807 } } peers { permanent_uuid: "c230911ad282421cabd56a069e80976f" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 43655 } }
I20260808 04:04:51.497577 16912 tablet_replica.cc:439] T 3085c84faa37435c9ec121620dd037f4 P 2013510f40c14486a71a3f2d3668d06a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2013510f40c14486a71a3f2d3668d06a" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 35837 } } peers { permanent_uuid: "442fa005e54743748ca0af7671c4a79a" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 39807 } } peers { permanent_uuid: "c230911ad282421cabd56a069e80976f" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 43655 } } }
I20260808 04:04:51.497879 16912 tablet_replica.cc:442] T 3085c84faa37435c9ec121620dd037f4 P 2013510f40c14486a71a3f2d3668d06a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 04:04:51.498160 16930 tablet_replica.cc:439] T 3085c84faa37435c9ec121620dd037f4 P 442fa005e54743748ca0af7671c4a79a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2013510f40c14486a71a3f2d3668d06a" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 35837 } } peers { permanent_uuid: "442fa005e54743748ca0af7671c4a79a" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 39807 } } peers { permanent_uuid: "c230911ad282421cabd56a069e80976f" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 43655 } } }
I20260808 04:04:51.498414 16930 tablet_replica.cc:442] T 3085c84faa37435c9ec121620dd037f4 P 442fa005e54743748ca0af7671c4a79a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 04:04:51.499435 16899 ts_tablet_manager.cc:1428] T 3085c84faa37435c9ec121620dd037f4 P 442fa005e54743748ca0af7671c4a79a: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260808 04:04:51.499420 16900 raft_consensus.cc:348] T 3085c84faa37435c9ec121620dd037f4 P c230911ad282421cabd56a069e80976f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2013510f40c14486a71a3f2d3668d06a" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 35837 } } peers { permanent_uuid: "442fa005e54743748ca0af7671c4a79a" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 39807 } } peers { permanent_uuid: "c230911ad282421cabd56a069e80976f" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 43655 } }
I20260808 04:04:51.499671 16900 raft_consensus.cc:374] T 3085c84faa37435c9ec121620dd037f4 P c230911ad282421cabd56a069e80976f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:04:51.499790 16900 raft_consensus.cc:729] T 3085c84faa37435c9ec121620dd037f4 P c230911ad282421cabd56a069e80976f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c230911ad282421cabd56a069e80976f, State: Initialized, Role: FOLLOWER
I20260808 04:04:51.500017 16900 consensus_queue.cc:260] T 3085c84faa37435c9ec121620dd037f4 P c230911ad282421cabd56a069e80976f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2013510f40c14486a71a3f2d3668d06a" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 35837 } } peers { permanent_uuid: "442fa005e54743748ca0af7671c4a79a" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 39807 } } peers { permanent_uuid: "c230911ad282421cabd56a069e80976f" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 43655 } }
I20260808 04:04:51.500562 16900 ts_tablet_manager.cc:1428] T 3085c84faa37435c9ec121620dd037f4 P c230911ad282421cabd56a069e80976f: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260808 04:04:51.500569 16913 tablet_replica.cc:439] T 3085c84faa37435c9ec121620dd037f4 P c230911ad282421cabd56a069e80976f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2013510f40c14486a71a3f2d3668d06a" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 35837 } } peers { permanent_uuid: "442fa005e54743748ca0af7671c4a79a" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 39807 } } peers { permanent_uuid: "c230911ad282421cabd56a069e80976f" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 43655 } } }
I20260808 04:04:51.501529 16913 tablet_replica.cc:442] T 3085c84faa37435c9ec121620dd037f4 P c230911ad282421cabd56a069e80976f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 04:04:51.504475 16902 ts_tablet_manager.cc:1428] T 3085c84faa37435c9ec121620dd037f4 P 2013510f40c14486a71a3f2d3668d06a: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20260808 04:04:51.610018 16930 raft_consensus.cc:482] T 3085c84faa37435c9ec121620dd037f4 P 442fa005e54743748ca0af7671c4a79a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:04:51.610239 16930 raft_consensus.cc:504] T 3085c84faa37435c9ec121620dd037f4 P 442fa005e54743748ca0af7671c4a79a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2013510f40c14486a71a3f2d3668d06a" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 35837 } } peers { permanent_uuid: "442fa005e54743748ca0af7671c4a79a" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 39807 } } peers { permanent_uuid: "c230911ad282421cabd56a069e80976f" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 43655 } }
I20260808 04:04:51.611104 16930 leader_election.cc:288] T 3085c84faa37435c9ec121620dd037f4 P 442fa005e54743748ca0af7671c4a79a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2013510f40c14486a71a3f2d3668d06a (127.5.83.1:35837), c230911ad282421cabd56a069e80976f (127.5.83.3:43655)
I20260808 04:04:51.611913 16355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3085c84faa37435c9ec121620dd037f4" candidate_uuid: "442fa005e54743748ca0af7671c4a79a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2013510f40c14486a71a3f2d3668d06a" is_pre_election: true
I20260808 04:04:51.611920 16781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3085c84faa37435c9ec121620dd037f4" candidate_uuid: "442fa005e54743748ca0af7671c4a79a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c230911ad282421cabd56a069e80976f" is_pre_election: true
I20260808 04:04:51.612202 16355 raft_consensus.cc:2454] T 3085c84faa37435c9ec121620dd037f4 P 2013510f40c14486a71a3f2d3668d06a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 442fa005e54743748ca0af7671c4a79a in term 0.
I20260808 04:04:51.612221 16781 raft_consensus.cc:2454] T 3085c84faa37435c9ec121620dd037f4 P c230911ad282421cabd56a069e80976f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 442fa005e54743748ca0af7671c4a79a in term 0.
I20260808 04:04:51.612682 16473 leader_election.cc:302] T 3085c84faa37435c9ec121620dd037f4 P 442fa005e54743748ca0af7671c4a79a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 442fa005e54743748ca0af7671c4a79a, c230911ad282421cabd56a069e80976f; no voters:
I20260808 04:04:51.612998 16930 raft_consensus.cc:2781] T 3085c84faa37435c9ec121620dd037f4 P 442fa005e54743748ca0af7671c4a79a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:04:51.613102 16930 raft_consensus.cc:482] T 3085c84faa37435c9ec121620dd037f4 P 442fa005e54743748ca0af7671c4a79a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:04:51.613206 16930 raft_consensus.cc:3037] T 3085c84faa37435c9ec121620dd037f4 P 442fa005e54743748ca0af7671c4a79a [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:04:51.615361 16930 raft_consensus.cc:504] T 3085c84faa37435c9ec121620dd037f4 P 442fa005e54743748ca0af7671c4a79a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2013510f40c14486a71a3f2d3668d06a" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 35837 } } peers { permanent_uuid: "442fa005e54743748ca0af7671c4a79a" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 39807 } } peers { permanent_uuid: "c230911ad282421cabd56a069e80976f" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 43655 } }
I20260808 04:04:51.615998 16930 leader_election.cc:288] T 3085c84faa37435c9ec121620dd037f4 P 442fa005e54743748ca0af7671c4a79a [CANDIDATE]: Term 1 election: Requested vote from peers 2013510f40c14486a71a3f2d3668d06a (127.5.83.1:35837), c230911ad282421cabd56a069e80976f (127.5.83.3:43655)
I20260808 04:04:51.616462 16781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3085c84faa37435c9ec121620dd037f4" candidate_uuid: "442fa005e54743748ca0af7671c4a79a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c230911ad282421cabd56a069e80976f"
I20260808 04:04:51.616477 16355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3085c84faa37435c9ec121620dd037f4" candidate_uuid: "442fa005e54743748ca0af7671c4a79a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2013510f40c14486a71a3f2d3668d06a"
I20260808 04:04:51.616691 16781 raft_consensus.cc:3037] T 3085c84faa37435c9ec121620dd037f4 P c230911ad282421cabd56a069e80976f [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:04:51.616691 16355 raft_consensus.cc:3037] T 3085c84faa37435c9ec121620dd037f4 P 2013510f40c14486a71a3f2d3668d06a [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:04:51.619547 16781 raft_consensus.cc:2454] T 3085c84faa37435c9ec121620dd037f4 P c230911ad282421cabd56a069e80976f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 442fa005e54743748ca0af7671c4a79a in term 1.
I20260808 04:04:51.619547 16355 raft_consensus.cc:2454] T 3085c84faa37435c9ec121620dd037f4 P 2013510f40c14486a71a3f2d3668d06a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 442fa005e54743748ca0af7671c4a79a in term 1.
I20260808 04:04:51.619977 16473 leader_election.cc:302] T 3085c84faa37435c9ec121620dd037f4 P 442fa005e54743748ca0af7671c4a79a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 442fa005e54743748ca0af7671c4a79a, c230911ad282421cabd56a069e80976f; no voters:
I20260808 04:04:51.620270 16930 raft_consensus.cc:2781] T 3085c84faa37435c9ec121620dd037f4 P 442fa005e54743748ca0af7671c4a79a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:04:51.620476 16930 raft_consensus.cc:686] T 3085c84faa37435c9ec121620dd037f4 P 442fa005e54743748ca0af7671c4a79a [term 1 LEADER]: Becoming Leader. State: Replica: 442fa005e54743748ca0af7671c4a79a, State: Running, Role: LEADER
I20260808 04:04:51.620759 16930 consensus_queue.cc:237] T 3085c84faa37435c9ec121620dd037f4 P 442fa005e54743748ca0af7671c4a79a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2013510f40c14486a71a3f2d3668d06a" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 35837 } } peers { permanent_uuid: "442fa005e54743748ca0af7671c4a79a" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 39807 } } peers { permanent_uuid: "c230911ad282421cabd56a069e80976f" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 43655 } }
I20260808 04:04:51.622632 16916 tablet_replica.cc:439] T 3085c84faa37435c9ec121620dd037f4 P 442fa005e54743748ca0af7671c4a79a: TxnStatusTablet state changed. Reason: New leader 442fa005e54743748ca0af7671c4a79a. Latest consensus state: current_term: 1 leader_uuid: "442fa005e54743748ca0af7671c4a79a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2013510f40c14486a71a3f2d3668d06a" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 35837 } } peers { permanent_uuid: "442fa005e54743748ca0af7671c4a79a" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 39807 } } peers { permanent_uuid: "c230911ad282421cabd56a069e80976f" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 43655 } } }
I20260808 04:04:51.622999 16916 tablet_replica.cc:442] T 3085c84faa37435c9ec121620dd037f4 P 442fa005e54743748ca0af7671c4a79a: This TxnStatusTablet replica's current role is: LEADER
I20260808 04:04:51.624222 16958 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260808 04:04:51.624496 16958 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260808 04:04:51.624298 16112 catalog_manager.cc:5495] T 3085c84faa37435c9ec121620dd037f4 P 442fa005e54743748ca0af7671c4a79a reported cstate change: term changed from 0 to 1, leader changed from <none> to 442fa005e54743748ca0af7671c4a79a (127.5.83.2). New cstate: current_term: 1 leader_uuid: "442fa005e54743748ca0af7671c4a79a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2013510f40c14486a71a3f2d3668d06a" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 35837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "442fa005e54743748ca0af7671c4a79a" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 39807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c230911ad282421cabd56a069e80976f" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 43655 } health_report { overall_health: UNKNOWN } } }
I20260808 04:04:51.649773 16890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:51.644838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34698 (local address 127.5.83.2:39807)
0808 04:04:51.645005 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:51.645011 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:51.645101 (+ 90us) server_negotiation.cc:407] Connection header received
0808 04:04:51.645346 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:51.645350 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:51.645437 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:51.645517 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:04:51.645887 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:51.646674 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:51.648002 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:51.648469 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:51.648593 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:51.648995 (+ 402us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:04:51.649118 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:04:51.649190 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:51.649352 (+ 162us) server_negotiation.cc:299] Negotiation successful
0808 04:04:51.649441 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 04:04:51.649755 16950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:51.644740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.2:39807 (local address 127.0.0.1:34698)
0808 04:04:51.645015 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:51.645034 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:51.645173 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:51.645558 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:51.645563 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:51.645575 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 04:04:51.645760 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:51.645769 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:51.646799 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:51.646805 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:51.647853 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:51.647865 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:51.648616 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:51.648622 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:51.648772 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:51.648825 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 04:04:51.649223 (+ 398us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 04:04:51.649244 (+ 21us) client_negotiation.cc:769] Sending connection context
0808 04:04:51.649320 (+ 76us) client_negotiation.cc:240] Negotiation successful
0808 04:04:51.649465 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260808 04:04:51.653502 16781 raft_consensus.cc:1261] T 3085c84faa37435c9ec121620dd037f4 P c230911ad282421cabd56a069e80976f [term 1 FOLLOWER]: Refusing update from remote peer 442fa005e54743748ca0af7671c4a79a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:04:51.653656 16355 raft_consensus.cc:1261] T 3085c84faa37435c9ec121620dd037f4 P 2013510f40c14486a71a3f2d3668d06a [term 1 FOLLOWER]: Refusing update from remote peer 442fa005e54743748ca0af7671c4a79a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:04:51.654152 16916 consensus_queue.cc:1035] T 3085c84faa37435c9ec121620dd037f4 P 442fa005e54743748ca0af7671c4a79a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c230911ad282421cabd56a069e80976f" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 43655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:04:51.654475 16930 consensus_queue.cc:1035] T 3085c84faa37435c9ec121620dd037f4 P 442fa005e54743748ca0af7671c4a79a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2013510f40c14486a71a3f2d3668d06a" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 35837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:04:51.657660 16912 tablet_replica.cc:439] T 3085c84faa37435c9ec121620dd037f4 P 2013510f40c14486a71a3f2d3668d06a: TxnStatusTablet state changed. Reason: New leader 442fa005e54743748ca0af7671c4a79a. Latest consensus state: current_term: 1 leader_uuid: "442fa005e54743748ca0af7671c4a79a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2013510f40c14486a71a3f2d3668d06a" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 35837 } } peers { permanent_uuid: "442fa005e54743748ca0af7671c4a79a" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 39807 } } peers { permanent_uuid: "c230911ad282421cabd56a069e80976f" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 43655 } } }
I20260808 04:04:51.657963 16912 tablet_replica.cc:442] T 3085c84faa37435c9ec121620dd037f4 P 2013510f40c14486a71a3f2d3668d06a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 04:04:51.659657 16930 tablet_replica.cc:439] T 3085c84faa37435c9ec121620dd037f4 P 442fa005e54743748ca0af7671c4a79a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "442fa005e54743748ca0af7671c4a79a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2013510f40c14486a71a3f2d3668d06a" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 35837 } } peers { permanent_uuid: "442fa005e54743748ca0af7671c4a79a" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 39807 } } peers { permanent_uuid: "c230911ad282421cabd56a069e80976f" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 43655 } } }
I20260808 04:04:51.660051 16930 tablet_replica.cc:442] T 3085c84faa37435c9ec121620dd037f4 P 442fa005e54743748ca0af7671c4a79a: This TxnStatusTablet replica's current role is: LEADER
I20260808 04:04:51.662197 16912 tablet_replica.cc:439] T 3085c84faa37435c9ec121620dd037f4 P 2013510f40c14486a71a3f2d3668d06a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "442fa005e54743748ca0af7671c4a79a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2013510f40c14486a71a3f2d3668d06a" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 35837 } } peers { permanent_uuid: "442fa005e54743748ca0af7671c4a79a" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 39807 } } peers { permanent_uuid: "c230911ad282421cabd56a069e80976f" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 43655 } } }
I20260808 04:04:51.662423 16912 tablet_replica.cc:442] T 3085c84faa37435c9ec121620dd037f4 P 2013510f40c14486a71a3f2d3668d06a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 04:04:51.664326 16913 tablet_replica.cc:439] T 3085c84faa37435c9ec121620dd037f4 P c230911ad282421cabd56a069e80976f: TxnStatusTablet state changed. Reason: New leader 442fa005e54743748ca0af7671c4a79a. Latest consensus state: current_term: 1 leader_uuid: "442fa005e54743748ca0af7671c4a79a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2013510f40c14486a71a3f2d3668d06a" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 35837 } } peers { permanent_uuid: "442fa005e54743748ca0af7671c4a79a" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 39807 } } peers { permanent_uuid: "c230911ad282421cabd56a069e80976f" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 43655 } } }
I20260808 04:04:51.664594 16913 tablet_replica.cc:442] T 3085c84faa37435c9ec121620dd037f4 P c230911ad282421cabd56a069e80976f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 04:04:51.665349 16911 tablet_replica.cc:439] T 3085c84faa37435c9ec121620dd037f4 P 442fa005e54743748ca0af7671c4a79a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "442fa005e54743748ca0af7671c4a79a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2013510f40c14486a71a3f2d3668d06a" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 35837 } } peers { permanent_uuid: "442fa005e54743748ca0af7671c4a79a" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 39807 } } peers { permanent_uuid: "c230911ad282421cabd56a069e80976f" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 43655 } } }
I20260808 04:04:51.665632 16911 tablet_replica.cc:442] T 3085c84faa37435c9ec121620dd037f4 P 442fa005e54743748ca0af7671c4a79a: This TxnStatusTablet replica's current role is: LEADER
I20260808 04:04:51.668452 16913 tablet_replica.cc:439] T 3085c84faa37435c9ec121620dd037f4 P c230911ad282421cabd56a069e80976f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "442fa005e54743748ca0af7671c4a79a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2013510f40c14486a71a3f2d3668d06a" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 35837 } } peers { permanent_uuid: "442fa005e54743748ca0af7671c4a79a" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 39807 } } peers { permanent_uuid: "c230911ad282421cabd56a069e80976f" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 43655 } } }
I20260808 04:04:51.668668 16913 tablet_replica.cc:442] T 3085c84faa37435c9ec121620dd037f4 P c230911ad282421cabd56a069e80976f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 04:04:51.683529 16890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:51.678499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.2:53525 (local address 127.5.83.2:39807)
0808 04:04:51.678686 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:51.678692 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:51.679007 (+ 315us) server_negotiation.cc:407] Connection header received
0808 04:04:51.679178 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:51.679183 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:51.679252 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:51.679334 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:04:51.679677 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:51.680503 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:51.681900 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:51.682343 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:51.682460 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:51.682840 (+ 380us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:04:51.682943 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:04:51.683015 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:51.683138 (+ 123us) server_negotiation.cc:299] Negotiation successful
0808 04:04:51.683205 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260808 04:04:51.683588 16968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:51.678418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.2:39807 (local address 127.5.83.2:53525)
0808 04:04:51.678918 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:51.678934 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:51.679063 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:51.679353 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:51.679357 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:51.679371 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 04:04:51.679561 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:51.679571 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:51.680625 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:51.680630 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:51.681767 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:51.681781 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:51.682479 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:51.682485 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:51.682630 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:51.682679 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 04:04:51.683032 (+ 353us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 04:04:51.683043 (+ 11us) client_negotiation.cc:769] Sending connection context
0808 04:04:51.683114 (+ 71us) client_negotiation.cc:240] Negotiation successful
0808 04:04:51.683242 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"mutex_wait_us":2,"thread_start_us":282,"threads_started":1}
I20260808 04:04:51.691784 16781 raft_consensus.cc:1261] T 543727dd35c145599e002e02592ba63e P c230911ad282421cabd56a069e80976f [term 1 FOLLOWER]: Refusing update from remote peer 442fa005e54743748ca0af7671c4a79a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:04:51.691823 16355 raft_consensus.cc:1261] T 543727dd35c145599e002e02592ba63e P 2013510f40c14486a71a3f2d3668d06a [term 1 FOLLOWER]: Refusing update from remote peer 442fa005e54743748ca0af7671c4a79a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:04:51.692315 16911 consensus_queue.cc:1035] T 543727dd35c145599e002e02592ba63e P 442fa005e54743748ca0af7671c4a79a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c230911ad282421cabd56a069e80976f" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 43655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:04:51.692824 16911 consensus_queue.cc:1035] T 543727dd35c145599e002e02592ba63e P 442fa005e54743748ca0af7671c4a79a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2013510f40c14486a71a3f2d3668d06a" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 35837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:04:51.749425 16108 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:04:51.749691 16108 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:04:51.752540 16108 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9f9dc08ed53644aa9bd23b443d2ae405: Sending DeleteTablet for 3 replicas of tablet 543727dd35c145599e002e02592ba63e
I20260808 04:04:51.753216 16313 tablet_service.cc:1514] Processing DeleteTablet for tablet 543727dd35c145599e002e02592ba63e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:04:51 UTC) from {username='oryx', principal='oryx/127.5.83.62@KRBTEST.COM'} at 127.0.0.1:33308
I20260808 04:04:51.753361 16547 tablet_service.cc:1514] Processing DeleteTablet for tablet 543727dd35c145599e002e02592ba63e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:04:51 UTC) from {username='oryx', principal='oryx/127.5.83.62@KRBTEST.COM'} at 127.0.0.1:34674
I20260808 04:04:51.753508 16933 tablet_replica.cc:330] T 543727dd35c145599e002e02592ba63e P 2013510f40c14486a71a3f2d3668d06a: stopping tablet replica
I20260808 04:04:51.753690 16933 raft_consensus.cc:2229] T 543727dd35c145599e002e02592ba63e P 2013510f40c14486a71a3f2d3668d06a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:04:51.753791 16935 tablet_replica.cc:330] T 543727dd35c145599e002e02592ba63e P 442fa005e54743748ca0af7671c4a79a: stopping tablet replica
I20260808 04:04:51.753870 16933 raft_consensus.cc:2258] T 543727dd35c145599e002e02592ba63e P 2013510f40c14486a71a3f2d3668d06a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:04:51.754007 16935 raft_consensus.cc:2229] T 543727dd35c145599e002e02592ba63e P 442fa005e54743748ca0af7671c4a79a [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:04:51.754338 16935 raft_consensus.cc:2258] T 543727dd35c145599e002e02592ba63e P 442fa005e54743748ca0af7671c4a79a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:04:51.754618 16752 tablet_service.cc:1514] Processing DeleteTablet for tablet 543727dd35c145599e002e02592ba63e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:04:51 UTC) from {username='oryx', principal='oryx/127.5.83.62@KRBTEST.COM'} at 127.0.0.1:41412
I20260808 04:04:51.754904 16936 tablet_replica.cc:330] T 543727dd35c145599e002e02592ba63e P c230911ad282421cabd56a069e80976f: stopping tablet replica
I20260808 04:04:51.755115 16936 raft_consensus.cc:2229] T 543727dd35c145599e002e02592ba63e P c230911ad282421cabd56a069e80976f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:04:51.755280 16936 raft_consensus.cc:2258] T 543727dd35c145599e002e02592ba63e P c230911ad282421cabd56a069e80976f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:04:51.756337 16936 ts_tablet_manager.cc:1905] T 543727dd35c145599e002e02592ba63e P c230911ad282421cabd56a069e80976f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:04:51.763262 16936 ts_tablet_manager.cc:1918] T 543727dd35c145599e002e02592ba63e P c230911ad282421cabd56a069e80976f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 04:04:51.763399 16936 log.cc:1196] T 543727dd35c145599e002e02592ba63e P c230911ad282421cabd56a069e80976f: Deleting WAL directory at /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/minicluster-data/ts-2/wal/wals/543727dd35c145599e002e02592ba63e
I20260808 04:04:51.763841 16936 ts_tablet_manager.cc:1939] T 543727dd35c145599e002e02592ba63e P c230911ad282421cabd56a069e80976f: Deleting consensus metadata
I20260808 04:04:51.764855 16088 catalog_manager.cc:4841] TS c230911ad282421cabd56a069e80976f (127.5.83.3:43655): tablet 543727dd35c145599e002e02592ba63e (table test-table [id=564ec0e52f7d4afda17f6e31e085683e]) successfully deleted
I20260808 04:04:51.765313 16933 ts_tablet_manager.cc:1905] T 543727dd35c145599e002e02592ba63e P 2013510f40c14486a71a3f2d3668d06a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:04:51.769984 16935 ts_tablet_manager.cc:1905] T 543727dd35c145599e002e02592ba63e P 442fa005e54743748ca0af7671c4a79a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:04:51.771695 16933 ts_tablet_manager.cc:1918] T 543727dd35c145599e002e02592ba63e P 2013510f40c14486a71a3f2d3668d06a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 04:04:51.771849 16933 log.cc:1196] T 543727dd35c145599e002e02592ba63e P 2013510f40c14486a71a3f2d3668d06a: Deleting WAL directory at /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/minicluster-data/ts-0/wal/wals/543727dd35c145599e002e02592ba63e
I20260808 04:04:51.772231 16933 ts_tablet_manager.cc:1939] T 543727dd35c145599e002e02592ba63e P 2013510f40c14486a71a3f2d3668d06a: Deleting consensus metadata
I20260808 04:04:51.773680 16086 catalog_manager.cc:4841] TS 2013510f40c14486a71a3f2d3668d06a (127.5.83.1:35837): tablet 543727dd35c145599e002e02592ba63e (table test-table [id=564ec0e52f7d4afda17f6e31e085683e]) successfully deleted
I20260808 04:04:51.778412 16935 ts_tablet_manager.cc:1918] T 543727dd35c145599e002e02592ba63e P 442fa005e54743748ca0af7671c4a79a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 04:04:51.778561 16935 log.cc:1196] T 543727dd35c145599e002e02592ba63e P 442fa005e54743748ca0af7671c4a79a: Deleting WAL directory at /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/minicluster-data/ts-1/wal/wals/543727dd35c145599e002e02592ba63e
I20260808 04:04:51.778939 16935 ts_tablet_manager.cc:1939] T 543727dd35c145599e002e02592ba63e P 442fa005e54743748ca0af7671c4a79a: Deleting consensus metadata
I20260808 04:04:51.779995 16087 catalog_manager.cc:4841] TS 442fa005e54743748ca0af7671c4a79a (127.5.83.2:39807): tablet 543727dd35c145599e002e02592ba63e (table test-table [id=564ec0e52f7d4afda17f6e31e085683e]) successfully deleted
Aug 08 04:04:51 dist-test-slave-1l3l krb5kdc[16047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161891, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 04:04:51 dist-test-slave-1l3l krb5kdc[16047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:04:51 dist-test-slave-1l3l krb5kdc[16047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
W20260808 04:04:51.814560 16254 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:04:51.804167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55486 (local address 127.5.83.62:33481)
0808 04:04:51.804370 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:51.804376 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:51.804748 (+ 372us) server_negotiation.cc:407] Connection header received
0808 04:04:51.804921 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:51.804934 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:51.805020 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:51.805136 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:51.806467 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:51.807598 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:51.808865 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:51.809547 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:51.809669 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:51.814298 (+ 4629us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55486: BlockingRecv error: recv got EOF from 127.0.0.1:55486 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 08 04:04:51 dist-test-slave-1l3l krb5kdc[16047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161891, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM
I20260808 04:04:51.831995 16254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:51.817420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55502 (local address 127.5.83.62:33481)
0808 04:04:51.817606 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:51.817611 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:51.817900 (+ 289us) server_negotiation.cc:407] Connection header received
0808 04:04:51.818073 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:51.818077 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:51.818144 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:51.818213 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:51.819654 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:51.820384 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:51.822236 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:51.822633 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:51.822731 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:51.827150 (+ 4419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:51.827155 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:51.827158 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:51.827188 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:51.829616 (+ 2428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:51.830323 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:51.830326 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:51.830328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:51.830458 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:51.830814 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:51.830817 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:51.830819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:51.831060 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:51.831243 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:51.831531 (+ 288us) server_negotiation.cc:299] Negotiation successful
0808 04:04:51.831685 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 04:04:51.835527 16108 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55502:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:04:51.836316 16108 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:04:51.842782 16313 tablet_service.cc:1467] Processing CreateTablet for tablet b49519423cf14a1284a2e96df4cceba9 (DEFAULT_TABLE table=test-table [id=1f639db750c644b29eacc81dcd4e82fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:04:51.843111 16752 tablet_service.cc:1467] Processing CreateTablet for tablet b49519423cf14a1284a2e96df4cceba9 (DEFAULT_TABLE table=test-table [id=1f639db750c644b29eacc81dcd4e82fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:04:51.843242 16313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b49519423cf14a1284a2e96df4cceba9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:04:51.843506 16752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b49519423cf14a1284a2e96df4cceba9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:04:51.849568 16902 tablet_bootstrap.cc:492] T b49519423cf14a1284a2e96df4cceba9 P 2013510f40c14486a71a3f2d3668d06a: Bootstrap starting.
I20260808 04:04:51.849586 16547 tablet_service.cc:1467] Processing CreateTablet for tablet b49519423cf14a1284a2e96df4cceba9 (DEFAULT_TABLE table=test-table [id=1f639db750c644b29eacc81dcd4e82fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:04:51.850003 16547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b49519423cf14a1284a2e96df4cceba9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:04:51.851197 16902 tablet_bootstrap.cc:654] T b49519423cf14a1284a2e96df4cceba9 P 2013510f40c14486a71a3f2d3668d06a: Neither blocks nor log segments found. Creating new log.
I20260808 04:04:51.853600 16902 tablet_bootstrap.cc:492] T b49519423cf14a1284a2e96df4cceba9 P 2013510f40c14486a71a3f2d3668d06a: No bootstrap required, opened a new log
I20260808 04:04:51.853729 16902 ts_tablet_manager.cc:1397] T b49519423cf14a1284a2e96df4cceba9 P 2013510f40c14486a71a3f2d3668d06a: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260808 04:04:51.854471 16902 raft_consensus.cc:348] T b49519423cf14a1284a2e96df4cceba9 P 2013510f40c14486a71a3f2d3668d06a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2013510f40c14486a71a3f2d3668d06a" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 35837 } } peers { permanent_uuid: "442fa005e54743748ca0af7671c4a79a" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 39807 } } peers { permanent_uuid: "c230911ad282421cabd56a069e80976f" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 43655 } }
I20260808 04:04:51.854710 16902 raft_consensus.cc:374] T b49519423cf14a1284a2e96df4cceba9 P 2013510f40c14486a71a3f2d3668d06a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:04:51.854794 16902 raft_consensus.cc:729] T b49519423cf14a1284a2e96df4cceba9 P 2013510f40c14486a71a3f2d3668d06a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2013510f40c14486a71a3f2d3668d06a, State: Initialized, Role: FOLLOWER
I20260808 04:04:51.854907 16899 tablet_bootstrap.cc:492] T b49519423cf14a1284a2e96df4cceba9 P 442fa005e54743748ca0af7671c4a79a: Bootstrap starting.
I20260808 04:04:51.855054 16902 consensus_queue.cc:260] T b49519423cf14a1284a2e96df4cceba9 P 2013510f40c14486a71a3f2d3668d06a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2013510f40c14486a71a3f2d3668d06a" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 35837 } } peers { permanent_uuid: "442fa005e54743748ca0af7671c4a79a" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 39807 } } peers { permanent_uuid: "c230911ad282421cabd56a069e80976f" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 43655 } }
I20260808 04:04:51.856523 16899 tablet_bootstrap.cc:654] T b49519423cf14a1284a2e96df4cceba9 P 442fa005e54743748ca0af7671c4a79a: Neither blocks nor log segments found. Creating new log.
I20260808 04:04:51.856765 16902 ts_tablet_manager.cc:1428] T b49519423cf14a1284a2e96df4cceba9 P 2013510f40c14486a71a3f2d3668d06a: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260808 04:04:51.857169 16900 tablet_bootstrap.cc:492] T b49519423cf14a1284a2e96df4cceba9 P c230911ad282421cabd56a069e80976f: Bootstrap starting.
I20260808 04:04:51.858781 16900 tablet_bootstrap.cc:654] T b49519423cf14a1284a2e96df4cceba9 P c230911ad282421cabd56a069e80976f: Neither blocks nor log segments found. Creating new log.
I20260808 04:04:51.859316 16899 tablet_bootstrap.cc:492] T b49519423cf14a1284a2e96df4cceba9 P 442fa005e54743748ca0af7671c4a79a: No bootstrap required, opened a new log
I20260808 04:04:51.859467 16899 ts_tablet_manager.cc:1397] T b49519423cf14a1284a2e96df4cceba9 P 442fa005e54743748ca0af7671c4a79a: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260808 04:04:51.860181 16899 raft_consensus.cc:348] T b49519423cf14a1284a2e96df4cceba9 P 442fa005e54743748ca0af7671c4a79a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2013510f40c14486a71a3f2d3668d06a" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 35837 } } peers { permanent_uuid: "442fa005e54743748ca0af7671c4a79a" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 39807 } } peers { permanent_uuid: "c230911ad282421cabd56a069e80976f" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 43655 } }
I20260808 04:04:51.860424 16899 raft_consensus.cc:374] T b49519423cf14a1284a2e96df4cceba9 P 442fa005e54743748ca0af7671c4a79a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:04:51.860530 16899 raft_consensus.cc:729] T b49519423cf14a1284a2e96df4cceba9 P 442fa005e54743748ca0af7671c4a79a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 442fa005e54743748ca0af7671c4a79a, State: Initialized, Role: FOLLOWER
I20260808 04:04:51.860774 16899 consensus_queue.cc:260] T b49519423cf14a1284a2e96df4cceba9 P 442fa005e54743748ca0af7671c4a79a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2013510f40c14486a71a3f2d3668d06a" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 35837 } } peers { permanent_uuid: "442fa005e54743748ca0af7671c4a79a" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 39807 } } peers { permanent_uuid: "c230911ad282421cabd56a069e80976f" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 43655 } }
I20260808 04:04:51.861249 16900 tablet_bootstrap.cc:492] T b49519423cf14a1284a2e96df4cceba9 P c230911ad282421cabd56a069e80976f: No bootstrap required, opened a new log
I20260808 04:04:51.861343 16899 ts_tablet_manager.cc:1428] T b49519423cf14a1284a2e96df4cceba9 P 442fa005e54743748ca0af7671c4a79a: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260808 04:04:51.861372 16900 ts_tablet_manager.cc:1397] T b49519423cf14a1284a2e96df4cceba9 P c230911ad282421cabd56a069e80976f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260808 04:04:51.862385 16900 raft_consensus.cc:348] T b49519423cf14a1284a2e96df4cceba9 P c230911ad282421cabd56a069e80976f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2013510f40c14486a71a3f2d3668d06a" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 35837 } } peers { permanent_uuid: "442fa005e54743748ca0af7671c4a79a" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 39807 } } peers { permanent_uuid: "c230911ad282421cabd56a069e80976f" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 43655 } }
I20260808 04:04:51.862627 16900 raft_consensus.cc:374] T b49519423cf14a1284a2e96df4cceba9 P c230911ad282421cabd56a069e80976f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:04:51.862720 16900 raft_consensus.cc:729] T b49519423cf14a1284a2e96df4cceba9 P c230911ad282421cabd56a069e80976f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c230911ad282421cabd56a069e80976f, State: Initialized, Role: FOLLOWER
I20260808 04:04:51.862941 16900 consensus_queue.cc:260] T b49519423cf14a1284a2e96df4cceba9 P c230911ad282421cabd56a069e80976f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2013510f40c14486a71a3f2d3668d06a" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 35837 } } peers { permanent_uuid: "442fa005e54743748ca0af7671c4a79a" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 39807 } } peers { permanent_uuid: "c230911ad282421cabd56a069e80976f" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 43655 } }
I20260808 04:04:51.863517 16900 ts_tablet_manager.cc:1428] T b49519423cf14a1284a2e96df4cceba9 P c230911ad282421cabd56a069e80976f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260808 04:04:52.087736 16913 raft_consensus.cc:482] T b49519423cf14a1284a2e96df4cceba9 P c230911ad282421cabd56a069e80976f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:04:52.087978 16913 raft_consensus.cc:504] T b49519423cf14a1284a2e96df4cceba9 P c230911ad282421cabd56a069e80976f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2013510f40c14486a71a3f2d3668d06a" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 35837 } } peers { permanent_uuid: "442fa005e54743748ca0af7671c4a79a" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 39807 } } peers { permanent_uuid: "c230911ad282421cabd56a069e80976f" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 43655 } }
I20260808 04:04:52.089565 16913 leader_election.cc:288] T b49519423cf14a1284a2e96df4cceba9 P c230911ad282421cabd56a069e80976f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2013510f40c14486a71a3f2d3668d06a (127.5.83.1:35837), 442fa005e54743748ca0af7671c4a79a (127.5.83.2:39807)
I20260808 04:04:52.108054 17000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:52.090112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.3:56083 (local address 127.5.83.1:35837)
0808 04:04:52.101095 (+ 10983us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:52.101101 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:52.101125 (+ 24us) server_negotiation.cc:407] Connection header received
0808 04:04:52.101201 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:52.101207 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:52.101281 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:52.101391 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:04:52.102190 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:52.103231 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:52.105798 (+ 2567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:52.107043 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:52.107174 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:52.107283 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:52.107585 (+ 302us) server_negotiation.cc:299] Negotiation successful
0808 04:04:52.107748 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10823,"thread_start_us":11012,"threads_started":1}
I20260808 04:04:52.108104 16999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:52.089837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:35837 (local address 127.5.83.3:56083)
0808 04:04:52.100824 (+ 10987us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:52.100850 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:52.101035 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:52.101596 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:52.101601 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:52.101633 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:04:52.102013 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:52.102025 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:52.103364 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:52.103370 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:52.105645 (+ 2275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:52.105659 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:52.107227 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:52.107233 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:52.107400 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:52.107439 (+ 39us) client_negotiation.cc:769] Sending connection context
0808 04:04:52.107540 (+ 101us) client_negotiation.cc:240] Negotiation successful
0808 04:04:52.107677 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10811,"mutex_wait_us":33,"thread_start_us":11554,"threads_started":1}
I20260808 04:04:52.108513 16890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:52.090299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.3:40389 (local address 127.5.83.2:39807)
0808 04:04:52.090515 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:52.090521 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:52.101762 (+ 11241us) server_negotiation.cc:407] Connection header received
0808 04:04:52.101999 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:52.102003 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:52.102065 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:52.102138 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:04:52.102682 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:52.103640 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:52.105714 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:52.107019 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:52.107174 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:52.107284 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:52.107816 (+ 532us) server_negotiation.cc:299] Negotiation successful
0808 04:04:52.107956 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 04:04:52.108871 16355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b49519423cf14a1284a2e96df4cceba9" candidate_uuid: "c230911ad282421cabd56a069e80976f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2013510f40c14486a71a3f2d3668d06a" is_pre_election: true
I20260808 04:04:52.109102 16355 raft_consensus.cc:2454] T b49519423cf14a1284a2e96df4cceba9 P 2013510f40c14486a71a3f2d3668d06a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c230911ad282421cabd56a069e80976f in term 0.
I20260808 04:04:52.109179 16997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:52.089866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.2:39807 (local address 127.5.83.3:40389)
0808 04:04:52.101636 (+ 11770us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:52.101651 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:52.101857 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:52.102199 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:52.102203 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:52.102226 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:04:52.102503 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:52.102513 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:52.103783 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:52.103789 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:52.105574 (+ 1785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:52.105596 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:52.107452 (+ 1856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:52.107457 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:52.107608 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:52.107641 (+ 33us) client_negotiation.cc:769] Sending connection context
0808 04:04:52.108693 (+ 1052us) client_negotiation.cc:240] Negotiation successful
0808 04:04:52.108838 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11654,"mutex_wait_us":5,"thread_start_us":10592,"threads_started":1}
I20260808 04:04:52.109663 16674 leader_election.cc:302] T b49519423cf14a1284a2e96df4cceba9 P c230911ad282421cabd56a069e80976f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2013510f40c14486a71a3f2d3668d06a, c230911ad282421cabd56a069e80976f; no voters:
I20260808 04:04:52.110086 16913 raft_consensus.cc:2781] T b49519423cf14a1284a2e96df4cceba9 P c230911ad282421cabd56a069e80976f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:04:52.110212 16913 raft_consensus.cc:482] T b49519423cf14a1284a2e96df4cceba9 P c230911ad282421cabd56a069e80976f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:04:52.110307 16913 raft_consensus.cc:3037] T b49519423cf14a1284a2e96df4cceba9 P c230911ad282421cabd56a069e80976f [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:04:52.110149 16584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b49519423cf14a1284a2e96df4cceba9" candidate_uuid: "c230911ad282421cabd56a069e80976f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "442fa005e54743748ca0af7671c4a79a" is_pre_election: true
I20260808 04:04:52.110636 16584 raft_consensus.cc:2454] T b49519423cf14a1284a2e96df4cceba9 P 442fa005e54743748ca0af7671c4a79a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c230911ad282421cabd56a069e80976f in term 0.
I20260808 04:04:52.112891 16913 raft_consensus.cc:504] T b49519423cf14a1284a2e96df4cceba9 P c230911ad282421cabd56a069e80976f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2013510f40c14486a71a3f2d3668d06a" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 35837 } } peers { permanent_uuid: "442fa005e54743748ca0af7671c4a79a" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 39807 } } peers { permanent_uuid: "c230911ad282421cabd56a069e80976f" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 43655 } }
I20260808 04:04:52.113493 16913 leader_election.cc:288] T b49519423cf14a1284a2e96df4cceba9 P c230911ad282421cabd56a069e80976f [CANDIDATE]: Term 1 election: Requested vote from peers 2013510f40c14486a71a3f2d3668d06a (127.5.83.1:35837), 442fa005e54743748ca0af7671c4a79a (127.5.83.2:39807)
I20260808 04:04:52.113914 16355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b49519423cf14a1284a2e96df4cceba9" candidate_uuid: "c230911ad282421cabd56a069e80976f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2013510f40c14486a71a3f2d3668d06a"
I20260808 04:04:52.114014 16584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b49519423cf14a1284a2e96df4cceba9" candidate_uuid: "c230911ad282421cabd56a069e80976f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "442fa005e54743748ca0af7671c4a79a"
I20260808 04:04:52.114099 16355 raft_consensus.cc:3037] T b49519423cf14a1284a2e96df4cceba9 P 2013510f40c14486a71a3f2d3668d06a [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:04:52.114208 16584 raft_consensus.cc:3037] T b49519423cf14a1284a2e96df4cceba9 P 442fa005e54743748ca0af7671c4a79a [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:04:52.116505 16355 raft_consensus.cc:2454] T b49519423cf14a1284a2e96df4cceba9 P 2013510f40c14486a71a3f2d3668d06a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c230911ad282421cabd56a069e80976f in term 1.
I20260808 04:04:52.116860 16674 leader_election.cc:302] T b49519423cf14a1284a2e96df4cceba9 P c230911ad282421cabd56a069e80976f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2013510f40c14486a71a3f2d3668d06a, c230911ad282421cabd56a069e80976f; no voters:
I20260808 04:04:52.117122 16913 raft_consensus.cc:2781] T b49519423cf14a1284a2e96df4cceba9 P c230911ad282421cabd56a069e80976f [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:04:52.117403 16584 raft_consensus.cc:2454] T b49519423cf14a1284a2e96df4cceba9 P 442fa005e54743748ca0af7671c4a79a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c230911ad282421cabd56a069e80976f in term 1.
I20260808 04:04:52.117533 16913 raft_consensus.cc:686] T b49519423cf14a1284a2e96df4cceba9 P c230911ad282421cabd56a069e80976f [term 1 LEADER]: Becoming Leader. State: Replica: c230911ad282421cabd56a069e80976f, State: Running, Role: LEADER
I20260808 04:04:52.117980 16913 consensus_queue.cc:237] T b49519423cf14a1284a2e96df4cceba9 P c230911ad282421cabd56a069e80976f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2013510f40c14486a71a3f2d3668d06a" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 35837 } } peers { permanent_uuid: "442fa005e54743748ca0af7671c4a79a" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 39807 } } peers { permanent_uuid: "c230911ad282421cabd56a069e80976f" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 43655 } }
I20260808 04:04:52.121656 16112 catalog_manager.cc:5495] T b49519423cf14a1284a2e96df4cceba9 P c230911ad282421cabd56a069e80976f reported cstate change: term changed from 0 to 1, leader changed from <none> to c230911ad282421cabd56a069e80976f (127.5.83.3). New cstate: current_term: 1 leader_uuid: "c230911ad282421cabd56a069e80976f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2013510f40c14486a71a3f2d3668d06a" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 35837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "442fa005e54743748ca0af7671c4a79a" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 39807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c230911ad282421cabd56a069e80976f" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 43655 } health_report { overall_health: HEALTHY } } }
I20260808 04:04:52.148223 17004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:52.142918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41416 (local address 127.5.83.3:43655)
0808 04:04:52.143499 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:52.143503 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:52.143523 (+ 20us) server_negotiation.cc:407] Connection header received
0808 04:04:52.143601 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:52.143605 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:52.143680 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:52.143836 (+ 156us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:04:52.144156 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:52.144947 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:52.146226 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:52.146643 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:52.146733 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:52.147101 (+ 368us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:04:52.147740 (+ 639us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:04:52.147858 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:52.147900 (+ 42us) server_negotiation.cc:299] Negotiation successful
0808 04:04:52.147962 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":321,"threads_started":1}
I20260808 04:04:52.151566 16355 raft_consensus.cc:1261] T b49519423cf14a1284a2e96df4cceba9 P 2013510f40c14486a71a3f2d3668d06a [term 1 FOLLOWER]: Refusing update from remote peer c230911ad282421cabd56a069e80976f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:04:52.151671 16584 raft_consensus.cc:1261] T b49519423cf14a1284a2e96df4cceba9 P 442fa005e54743748ca0af7671c4a79a [term 1 FOLLOWER]: Refusing update from remote peer c230911ad282421cabd56a069e80976f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:04:52.152252 17001 consensus_queue.cc:1035] T b49519423cf14a1284a2e96df4cceba9 P c230911ad282421cabd56a069e80976f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2013510f40c14486a71a3f2d3668d06a" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 35837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:04:52.152624 16913 consensus_queue.cc:1035] T b49519423cf14a1284a2e96df4cceba9 P c230911ad282421cabd56a069e80976f [LEADER]: Connected to new peer: Peer: permanent_uuid: "442fa005e54743748ca0af7671c4a79a" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 39807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:04:52.157145 16922 mvcc.cc:205] Tried to move back new op lower bound from 7316119110246260736 to 7316119110119878656. Current Snapshot: MvccSnapshot[applied={T|T < 7316119110246260736}]
I20260808 04:04:52.168507 16112 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:04:52.168749 16112 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:04:52.171607 16112 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9f9dc08ed53644aa9bd23b443d2ae405: Sending DeleteTablet for 3 replicas of tablet b49519423cf14a1284a2e96df4cceba9
I20260808 04:04:52.172363 16313 tablet_service.cc:1514] Processing DeleteTablet for tablet b49519423cf14a1284a2e96df4cceba9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:04:52 UTC) from {username='oryx', principal='oryx/127.5.83.62@KRBTEST.COM'} at 127.0.0.1:33308
I20260808 04:04:52.172609 16547 tablet_service.cc:1514] Processing DeleteTablet for tablet b49519423cf14a1284a2e96df4cceba9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:04:52 UTC) from {username='oryx', principal='oryx/127.5.83.62@KRBTEST.COM'} at 127.0.0.1:34674
I20260808 04:04:52.172901 16935 tablet_replica.cc:330] T b49519423cf14a1284a2e96df4cceba9 P 442fa005e54743748ca0af7671c4a79a: stopping tablet replica
I20260808 04:04:52.173076 16935 raft_consensus.cc:2229] T b49519423cf14a1284a2e96df4cceba9 P 442fa005e54743748ca0af7671c4a79a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:04:52.173100 16933 tablet_replica.cc:330] T b49519423cf14a1284a2e96df4cceba9 P 2013510f40c14486a71a3f2d3668d06a: stopping tablet replica
I20260808 04:04:52.173277 16933 raft_consensus.cc:2229] T b49519423cf14a1284a2e96df4cceba9 P 2013510f40c14486a71a3f2d3668d06a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:04:52.173295 16935 raft_consensus.cc:2258] T b49519423cf14a1284a2e96df4cceba9 P 442fa005e54743748ca0af7671c4a79a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:04:52.173522 16933 raft_consensus.cc:2258] T b49519423cf14a1284a2e96df4cceba9 P 2013510f40c14486a71a3f2d3668d06a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:04:52.173802 16752 tablet_service.cc:1514] Processing DeleteTablet for tablet b49519423cf14a1284a2e96df4cceba9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:04:52 UTC) from {username='oryx', principal='oryx/127.5.83.62@KRBTEST.COM'} at 127.0.0.1:41412
I20260808 04:04:52.173755 16112 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55502:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260808 04:04:52.174064 16936 tablet_replica.cc:330] T b49519423cf14a1284a2e96df4cceba9 P c230911ad282421cabd56a069e80976f: stopping tablet replica
I20260808 04:04:52.174243 16936 raft_consensus.cc:2229] T b49519423cf14a1284a2e96df4cceba9 P c230911ad282421cabd56a069e80976f [term 1 LEADER]: Raft consensus shutting down.
W20260808 04:04:52.174348 16112 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:04:52.174717 16936 raft_consensus.cc:2258] T b49519423cf14a1284a2e96df4cceba9 P c230911ad282421cabd56a069e80976f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:04:52.175446 16933 ts_tablet_manager.cc:1905] T b49519423cf14a1284a2e96df4cceba9 P 2013510f40c14486a71a3f2d3668d06a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:04:52.175863 16936 ts_tablet_manager.cc:1905] T b49519423cf14a1284a2e96df4cceba9 P c230911ad282421cabd56a069e80976f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:04:52.180517 16935 ts_tablet_manager.cc:1905] T b49519423cf14a1284a2e96df4cceba9 P 442fa005e54743748ca0af7671c4a79a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:04:52.183250 16936 ts_tablet_manager.cc:1918] T b49519423cf14a1284a2e96df4cceba9 P c230911ad282421cabd56a069e80976f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:04:52.183387 16936 log.cc:1196] T b49519423cf14a1284a2e96df4cceba9 P c230911ad282421cabd56a069e80976f: Deleting WAL directory at /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/minicluster-data/ts-2/wal/wals/b49519423cf14a1284a2e96df4cceba9
I20260808 04:04:52.183862 16936 ts_tablet_manager.cc:1939] T b49519423cf14a1284a2e96df4cceba9 P c230911ad282421cabd56a069e80976f: Deleting consensus metadata
I20260808 04:04:52.184921 16088 catalog_manager.cc:4841] TS c230911ad282421cabd56a069e80976f (127.5.83.3:43655): tablet b49519423cf14a1284a2e96df4cceba9 (table test-table [id=1f639db750c644b29eacc81dcd4e82fb]) successfully deleted
I20260808 04:04:52.186225 16933 ts_tablet_manager.cc:1918] T b49519423cf14a1284a2e96df4cceba9 P 2013510f40c14486a71a3f2d3668d06a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:04:52.186437 16933 log.cc:1196] T b49519423cf14a1284a2e96df4cceba9 P 2013510f40c14486a71a3f2d3668d06a: Deleting WAL directory at /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/minicluster-data/ts-0/wal/wals/b49519423cf14a1284a2e96df4cceba9
I20260808 04:04:52.186971 16933 ts_tablet_manager.cc:1939] T b49519423cf14a1284a2e96df4cceba9 P 2013510f40c14486a71a3f2d3668d06a: Deleting consensus metadata
I20260808 04:04:52.188248 16086 catalog_manager.cc:4841] TS 2013510f40c14486a71a3f2d3668d06a (127.5.83.1:35837): tablet b49519423cf14a1284a2e96df4cceba9 (table test-table [id=1f639db750c644b29eacc81dcd4e82fb]) successfully deleted
I20260808 04:04:52.189241 16313 tablet_service.cc:1467] Processing CreateTablet for tablet 6de7c77b84e6426aa8369ae8666af84f (DEFAULT_TABLE table=test-table [id=9f8f24289b304d96b7dac8f6bfb03421]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:04:52.189481 16547 tablet_service.cc:1467] Processing CreateTablet for tablet 6de7c77b84e6426aa8369ae8666af84f (DEFAULT_TABLE table=test-table [id=9f8f24289b304d96b7dac8f6bfb03421]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:04:52.189643 16313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6de7c77b84e6426aa8369ae8666af84f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:04:52.189832 16547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6de7c77b84e6426aa8369ae8666af84f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:04:52.195773 16899 tablet_bootstrap.cc:492] T 6de7c77b84e6426aa8369ae8666af84f P 442fa005e54743748ca0af7671c4a79a: Bootstrap starting.
I20260808 04:04:52.196458 16752 tablet_service.cc:1467] Processing CreateTablet for tablet 6de7c77b84e6426aa8369ae8666af84f (DEFAULT_TABLE table=test-table [id=9f8f24289b304d96b7dac8f6bfb03421]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:04:52.196556 16902 tablet_bootstrap.cc:492] T 6de7c77b84e6426aa8369ae8666af84f P 2013510f40c14486a71a3f2d3668d06a: Bootstrap starting.
I20260808 04:04:52.196858 16752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6de7c77b84e6426aa8369ae8666af84f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:04:52.197410 16899 tablet_bootstrap.cc:654] T 6de7c77b84e6426aa8369ae8666af84f P 442fa005e54743748ca0af7671c4a79a: Neither blocks nor log segments found. Creating new log.
I20260808 04:04:52.198037 16902 tablet_bootstrap.cc:654] T 6de7c77b84e6426aa8369ae8666af84f P 2013510f40c14486a71a3f2d3668d06a: Neither blocks nor log segments found. Creating new log.
I20260808 04:04:52.205727 16935 ts_tablet_manager.cc:1918] T b49519423cf14a1284a2e96df4cceba9 P 442fa005e54743748ca0af7671c4a79a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:04:52.205863 16935 log.cc:1196] T b49519423cf14a1284a2e96df4cceba9 P 442fa005e54743748ca0af7671c4a79a: Deleting WAL directory at /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/minicluster-data/ts-1/wal/wals/b49519423cf14a1284a2e96df4cceba9
I20260808 04:04:52.206295 16935 ts_tablet_manager.cc:1939] T b49519423cf14a1284a2e96df4cceba9 P 442fa005e54743748ca0af7671c4a79a: Deleting consensus metadata
I20260808 04:04:52.206602 16900 tablet_bootstrap.cc:492] T 6de7c77b84e6426aa8369ae8666af84f P c230911ad282421cabd56a069e80976f: Bootstrap starting.
I20260808 04:04:52.208350 16900 tablet_bootstrap.cc:654] T 6de7c77b84e6426aa8369ae8666af84f P c230911ad282421cabd56a069e80976f: Neither blocks nor log segments found. Creating new log.
I20260808 04:04:52.210194 16902 tablet_bootstrap.cc:492] T 6de7c77b84e6426aa8369ae8666af84f P 2013510f40c14486a71a3f2d3668d06a: No bootstrap required, opened a new log
I20260808 04:04:52.210384 16902 ts_tablet_manager.cc:1397] T 6de7c77b84e6426aa8369ae8666af84f P 2013510f40c14486a71a3f2d3668d06a: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.008s
I20260808 04:04:52.211259 16902 raft_consensus.cc:348] T 6de7c77b84e6426aa8369ae8666af84f P 2013510f40c14486a71a3f2d3668d06a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2013510f40c14486a71a3f2d3668d06a" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 35837 } } peers { permanent_uuid: "442fa005e54743748ca0af7671c4a79a" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 39807 } } peers { permanent_uuid: "c230911ad282421cabd56a069e80976f" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 43655 } }
I20260808 04:04:52.211537 16902 raft_consensus.cc:374] T 6de7c77b84e6426aa8369ae8666af84f P 2013510f40c14486a71a3f2d3668d06a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:04:52.211625 16902 raft_consensus.cc:729] T 6de7c77b84e6426aa8369ae8666af84f P 2013510f40c14486a71a3f2d3668d06a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2013510f40c14486a71a3f2d3668d06a, State: Initialized, Role: FOLLOWER
I20260808 04:04:52.212013 16900 tablet_bootstrap.cc:492] T 6de7c77b84e6426aa8369ae8666af84f P c230911ad282421cabd56a069e80976f: No bootstrap required, opened a new log
I20260808 04:04:52.211948 16902 consensus_queue.cc:260] T 6de7c77b84e6426aa8369ae8666af84f P 2013510f40c14486a71a3f2d3668d06a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2013510f40c14486a71a3f2d3668d06a" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 35837 } } peers { permanent_uuid: "442fa005e54743748ca0af7671c4a79a" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 39807 } } peers { permanent_uuid: "c230911ad282421cabd56a069e80976f" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 43655 } }
I20260808 04:04:52.212180 16900 ts_tablet_manager.cc:1397] T 6de7c77b84e6426aa8369ae8666af84f P c230911ad282421cabd56a069e80976f: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260808 04:04:52.212150 16087 catalog_manager.cc:4841] TS 442fa005e54743748ca0af7671c4a79a (127.5.83.2:39807): tablet b49519423cf14a1284a2e96df4cceba9 (table test-table [id=1f639db750c644b29eacc81dcd4e82fb]) successfully deleted
I20260808 04:04:52.212966 16902 ts_tablet_manager.cc:1428] T 6de7c77b84e6426aa8369ae8666af84f P 2013510f40c14486a71a3f2d3668d06a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260808 04:04:52.212981 16900 raft_consensus.cc:348] T 6de7c77b84e6426aa8369ae8666af84f P c230911ad282421cabd56a069e80976f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2013510f40c14486a71a3f2d3668d06a" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 35837 } } peers { permanent_uuid: "442fa005e54743748ca0af7671c4a79a" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 39807 } } peers { permanent_uuid: "c230911ad282421cabd56a069e80976f" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 43655 } }
I20260808 04:04:52.213249 16900 raft_consensus.cc:374] T 6de7c77b84e6426aa8369ae8666af84f P c230911ad282421cabd56a069e80976f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:04:52.213335 16900 raft_consensus.cc:729] T 6de7c77b84e6426aa8369ae8666af84f P c230911ad282421cabd56a069e80976f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c230911ad282421cabd56a069e80976f, State: Initialized, Role: FOLLOWER
I20260808 04:04:52.213567 16900 consensus_queue.cc:260] T 6de7c77b84e6426aa8369ae8666af84f P c230911ad282421cabd56a069e80976f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2013510f40c14486a71a3f2d3668d06a" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 35837 } } peers { permanent_uuid: "442fa005e54743748ca0af7671c4a79a" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 39807 } } peers { permanent_uuid: "c230911ad282421cabd56a069e80976f" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 43655 } }
I20260808 04:04:52.214502 16900 ts_tablet_manager.cc:1428] T 6de7c77b84e6426aa8369ae8666af84f P c230911ad282421cabd56a069e80976f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 04:04:52.216254 16899 tablet_bootstrap.cc:492] T 6de7c77b84e6426aa8369ae8666af84f P 442fa005e54743748ca0af7671c4a79a: No bootstrap required, opened a new log
I20260808 04:04:52.216470 16899 ts_tablet_manager.cc:1397] T 6de7c77b84e6426aa8369ae8666af84f P 442fa005e54743748ca0af7671c4a79a: Time spent bootstrapping tablet: real 0.021s user 0.009s sys 0.003s
I20260808 04:04:52.217226 16899 raft_consensus.cc:348] T 6de7c77b84e6426aa8369ae8666af84f P 442fa005e54743748ca0af7671c4a79a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2013510f40c14486a71a3f2d3668d06a" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 35837 } } peers { permanent_uuid: "442fa005e54743748ca0af7671c4a79a" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 39807 } } peers { permanent_uuid: "c230911ad282421cabd56a069e80976f" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 43655 } }
I20260808 04:04:52.217499 16899 raft_consensus.cc:374] T 6de7c77b84e6426aa8369ae8666af84f P 442fa005e54743748ca0af7671c4a79a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:04:52.217612 16899 raft_consensus.cc:729] T 6de7c77b84e6426aa8369ae8666af84f P 442fa005e54743748ca0af7671c4a79a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 442fa005e54743748ca0af7671c4a79a, State: Initialized, Role: FOLLOWER
I20260808 04:04:52.217918 16899 consensus_queue.cc:260] T 6de7c77b84e6426aa8369ae8666af84f P 442fa005e54743748ca0af7671c4a79a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2013510f40c14486a71a3f2d3668d06a" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 35837 } } peers { permanent_uuid: "442fa005e54743748ca0af7671c4a79a" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 39807 } } peers { permanent_uuid: "c230911ad282421cabd56a069e80976f" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 43655 } }
I20260808 04:04:52.218508 16899 ts_tablet_manager.cc:1428] T 6de7c77b84e6426aa8369ae8666af84f P 442fa005e54743748ca0af7671c4a79a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 04:04:52.257130 16916 raft_consensus.cc:482] T 6de7c77b84e6426aa8369ae8666af84f P 442fa005e54743748ca0af7671c4a79a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:04:52.257352 16916 raft_consensus.cc:504] T 6de7c77b84e6426aa8369ae8666af84f P 442fa005e54743748ca0af7671c4a79a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2013510f40c14486a71a3f2d3668d06a" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 35837 } } peers { permanent_uuid: "442fa005e54743748ca0af7671c4a79a" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 39807 } } peers { permanent_uuid: "c230911ad282421cabd56a069e80976f" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 43655 } }
I20260808 04:04:52.258056 16916 leader_election.cc:288] T 6de7c77b84e6426aa8369ae8666af84f P 442fa005e54743748ca0af7671c4a79a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2013510f40c14486a71a3f2d3668d06a (127.5.83.1:35837), c230911ad282421cabd56a069e80976f (127.5.83.3:43655)
I20260808 04:04:52.258590 16355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6de7c77b84e6426aa8369ae8666af84f" candidate_uuid: "442fa005e54743748ca0af7671c4a79a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2013510f40c14486a71a3f2d3668d06a" is_pre_election: true
I20260808 04:04:52.258853 16355 raft_consensus.cc:2454] T 6de7c77b84e6426aa8369ae8666af84f P 2013510f40c14486a71a3f2d3668d06a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 442fa005e54743748ca0af7671c4a79a in term 0.
I20260808 04:04:52.258797 16781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6de7c77b84e6426aa8369ae8666af84f" candidate_uuid: "442fa005e54743748ca0af7671c4a79a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c230911ad282421cabd56a069e80976f" is_pre_election: true
I20260808 04:04:52.259078 16781 raft_consensus.cc:2454] T 6de7c77b84e6426aa8369ae8666af84f P c230911ad282421cabd56a069e80976f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 442fa005e54743748ca0af7671c4a79a in term 0.
I20260808 04:04:52.259272 16470 leader_election.cc:302] T 6de7c77b84e6426aa8369ae8666af84f P 442fa005e54743748ca0af7671c4a79a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2013510f40c14486a71a3f2d3668d06a, 442fa005e54743748ca0af7671c4a79a; no voters:
I20260808 04:04:52.259567 16916 raft_consensus.cc:2781] T 6de7c77b84e6426aa8369ae8666af84f P 442fa005e54743748ca0af7671c4a79a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:04:52.259689 16916 raft_consensus.cc:482] T 6de7c77b84e6426aa8369ae8666af84f P 442fa005e54743748ca0af7671c4a79a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:04:52.259805 16916 raft_consensus.cc:3037] T 6de7c77b84e6426aa8369ae8666af84f P 442fa005e54743748ca0af7671c4a79a [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:04:52.262698 16916 raft_consensus.cc:504] T 6de7c77b84e6426aa8369ae8666af84f P 442fa005e54743748ca0af7671c4a79a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2013510f40c14486a71a3f2d3668d06a" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 35837 } } peers { permanent_uuid: "442fa005e54743748ca0af7671c4a79a" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 39807 } } peers { permanent_uuid: "c230911ad282421cabd56a069e80976f" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 43655 } }
I20260808 04:04:52.263311 16916 leader_election.cc:288] T 6de7c77b84e6426aa8369ae8666af84f P 442fa005e54743748ca0af7671c4a79a [CANDIDATE]: Term 1 election: Requested vote from peers 2013510f40c14486a71a3f2d3668d06a (127.5.83.1:35837), c230911ad282421cabd56a069e80976f (127.5.83.3:43655)
I20260808 04:04:52.263783 16781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6de7c77b84e6426aa8369ae8666af84f" candidate_uuid: "442fa005e54743748ca0af7671c4a79a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c230911ad282421cabd56a069e80976f"
I20260808 04:04:52.263797 16355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6de7c77b84e6426aa8369ae8666af84f" candidate_uuid: "442fa005e54743748ca0af7671c4a79a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2013510f40c14486a71a3f2d3668d06a"
I20260808 04:04:52.263968 16781 raft_consensus.cc:3037] T 6de7c77b84e6426aa8369ae8666af84f P c230911ad282421cabd56a069e80976f [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:04:52.263994 16355 raft_consensus.cc:3037] T 6de7c77b84e6426aa8369ae8666af84f P 2013510f40c14486a71a3f2d3668d06a [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:04:52.266417 16781 raft_consensus.cc:2454] T 6de7c77b84e6426aa8369ae8666af84f P c230911ad282421cabd56a069e80976f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 442fa005e54743748ca0af7671c4a79a in term 1.
I20260808 04:04:52.266798 16473 leader_election.cc:302] T 6de7c77b84e6426aa8369ae8666af84f P 442fa005e54743748ca0af7671c4a79a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 442fa005e54743748ca0af7671c4a79a, c230911ad282421cabd56a069e80976f; no voters:
I20260808 04:04:52.266980 16355 raft_consensus.cc:2454] T 6de7c77b84e6426aa8369ae8666af84f P 2013510f40c14486a71a3f2d3668d06a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 442fa005e54743748ca0af7671c4a79a in term 1.
I20260808 04:04:52.267078 16916 raft_consensus.cc:2781] T 6de7c77b84e6426aa8369ae8666af84f P 442fa005e54743748ca0af7671c4a79a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:04:52.267554 16916 raft_consensus.cc:686] T 6de7c77b84e6426aa8369ae8666af84f P 442fa005e54743748ca0af7671c4a79a [term 1 LEADER]: Becoming Leader. State: Replica: 442fa005e54743748ca0af7671c4a79a, State: Running, Role: LEADER
I20260808 04:04:52.268023 16916 consensus_queue.cc:237] T 6de7c77b84e6426aa8369ae8666af84f P 442fa005e54743748ca0af7671c4a79a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2013510f40c14486a71a3f2d3668d06a" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 35837 } } peers { permanent_uuid: "442fa005e54743748ca0af7671c4a79a" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 39807 } } peers { permanent_uuid: "c230911ad282421cabd56a069e80976f" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 43655 } }
I20260808 04:04:52.270427 16112 catalog_manager.cc:5495] T 6de7c77b84e6426aa8369ae8666af84f P 442fa005e54743748ca0af7671c4a79a reported cstate change: term changed from 0 to 1, leader changed from <none> to 442fa005e54743748ca0af7671c4a79a (127.5.83.2). New cstate: current_term: 1 leader_uuid: "442fa005e54743748ca0af7671c4a79a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2013510f40c14486a71a3f2d3668d06a" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 35837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "442fa005e54743748ca0af7671c4a79a" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 39807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c230911ad282421cabd56a069e80976f" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 43655 } health_report { overall_health: UNKNOWN } } }
I20260808 04:04:52.291915 16890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:52.286891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34702 (local address 127.5.83.2:39807)
0808 04:04:52.287050 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:52.287056 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:52.287308 (+ 252us) server_negotiation.cc:407] Connection header received
0808 04:04:52.287502 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:52.287508 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:52.287574 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:52.287646 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:04:52.288048 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:52.288852 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:52.290129 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:52.290622 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:52.290726 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:52.291199 (+ 473us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:04:52.291311 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:04:52.291386 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:52.291531 (+ 145us) server_negotiation.cc:299] Negotiation successful
0808 04:04:52.291609 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260808 04:04:52.300311 16355 raft_consensus.cc:1261] T 6de7c77b84e6426aa8369ae8666af84f P 2013510f40c14486a71a3f2d3668d06a [term 1 FOLLOWER]: Refusing update from remote peer 442fa005e54743748ca0af7671c4a79a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:04:52.300768 16916 consensus_queue.cc:1035] T 6de7c77b84e6426aa8369ae8666af84f P 442fa005e54743748ca0af7671c4a79a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2013510f40c14486a71a3f2d3668d06a" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 35837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:04:52.301429 16781 raft_consensus.cc:1261] T 6de7c77b84e6426aa8369ae8666af84f P c230911ad282421cabd56a069e80976f [term 1 FOLLOWER]: Refusing update from remote peer 442fa005e54743748ca0af7671c4a79a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:04:52.303052 16916 consensus_queue.cc:1035] T 6de7c77b84e6426aa8369ae8666af84f P 442fa005e54743748ca0af7671c4a79a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c230911ad282421cabd56a069e80976f" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 43655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 04:04:52.374135 16108 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:04:52.374352 16108 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:04:52.377135 16108 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9f9dc08ed53644aa9bd23b443d2ae405: Sending DeleteTablet for 3 replicas of tablet 6de7c77b84e6426aa8369ae8666af84f
I20260808 04:04:52.377892 16313 tablet_service.cc:1514] Processing DeleteTablet for tablet 6de7c77b84e6426aa8369ae8666af84f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:04:52 UTC) from {username='oryx', principal='oryx/127.5.83.62@KRBTEST.COM'} at 127.0.0.1:33308
I20260808 04:04:52.378094 16547 tablet_service.cc:1514] Processing DeleteTablet for tablet 6de7c77b84e6426aa8369ae8666af84f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:04:52 UTC) from {username='oryx', principal='oryx/127.5.83.62@KRBTEST.COM'} at 127.0.0.1:34674
I20260808 04:04:52.378219 16752 tablet_service.cc:1514] Processing DeleteTablet for tablet 6de7c77b84e6426aa8369ae8666af84f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:04:52 UTC) from {username='oryx', principal='oryx/127.5.83.62@KRBTEST.COM'} at 127.0.0.1:41412
I20260808 04:04:52.378405 16933 tablet_replica.cc:330] T 6de7c77b84e6426aa8369ae8666af84f P 2013510f40c14486a71a3f2d3668d06a: stopping tablet replica
I20260808 04:04:52.378520 16936 tablet_replica.cc:330] T 6de7c77b84e6426aa8369ae8666af84f P c230911ad282421cabd56a069e80976f: stopping tablet replica
I20260808 04:04:52.378623 16933 raft_consensus.cc:2229] T 6de7c77b84e6426aa8369ae8666af84f P 2013510f40c14486a71a3f2d3668d06a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:04:52.378732 16936 raft_consensus.cc:2229] T 6de7c77b84e6426aa8369ae8666af84f P c230911ad282421cabd56a069e80976f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:04:52.378815 16933 raft_consensus.cc:2258] T 6de7c77b84e6426aa8369ae8666af84f P 2013510f40c14486a71a3f2d3668d06a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:04:52.378931 16936 raft_consensus.cc:2258] T 6de7c77b84e6426aa8369ae8666af84f P c230911ad282421cabd56a069e80976f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:04:52.379099 16935 tablet_replica.cc:330] T 6de7c77b84e6426aa8369ae8666af84f P 442fa005e54743748ca0af7671c4a79a: stopping tablet replica
I20260808 04:04:52.379298 16935 raft_consensus.cc:2229] T 6de7c77b84e6426aa8369ae8666af84f P 442fa005e54743748ca0af7671c4a79a [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:04:52.379621 16935 raft_consensus.cc:2258] T 6de7c77b84e6426aa8369ae8666af84f P 442fa005e54743748ca0af7671c4a79a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:04:52.380781 16933 ts_tablet_manager.cc:1905] T 6de7c77b84e6426aa8369ae8666af84f P 2013510f40c14486a71a3f2d3668d06a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:04:52.380817 16936 ts_tablet_manager.cc:1905] T 6de7c77b84e6426aa8369ae8666af84f P c230911ad282421cabd56a069e80976f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:04:52.384680 16935 ts_tablet_manager.cc:1905] T 6de7c77b84e6426aa8369ae8666af84f P 442fa005e54743748ca0af7671c4a79a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:04:52.385185 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 16224
I20260808 04:04:52.385950 16933 ts_tablet_manager.cc:1918] T 6de7c77b84e6426aa8369ae8666af84f P 2013510f40c14486a71a3f2d3668d06a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 04:04:52.386091 16933 log.cc:1196] T 6de7c77b84e6426aa8369ae8666af84f P 2013510f40c14486a71a3f2d3668d06a: Deleting WAL directory at /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/minicluster-data/ts-0/wal/wals/6de7c77b84e6426aa8369ae8666af84f
I20260808 04:04:52.386505 16933 ts_tablet_manager.cc:1939] T 6de7c77b84e6426aa8369ae8666af84f P 2013510f40c14486a71a3f2d3668d06a: Deleting consensus metadata
I20260808 04:04:52.387693 16086 catalog_manager.cc:4841] TS 2013510f40c14486a71a3f2d3668d06a (127.5.83.1:35837): tablet 6de7c77b84e6426aa8369ae8666af84f (table test-table [id=9f8f24289b304d96b7dac8f6bfb03421]) successfully deleted
I20260808 04:04:52.391029 16935 ts_tablet_manager.cc:1918] T 6de7c77b84e6426aa8369ae8666af84f P 442fa005e54743748ca0af7671c4a79a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 04:04:52.391156 16935 log.cc:1196] T 6de7c77b84e6426aa8369ae8666af84f P 442fa005e54743748ca0af7671c4a79a: Deleting WAL directory at /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/minicluster-data/ts-1/wal/wals/6de7c77b84e6426aa8369ae8666af84f
I20260808 04:04:52.391532 16935 ts_tablet_manager.cc:1939] T 6de7c77b84e6426aa8369ae8666af84f P 442fa005e54743748ca0af7671c4a79a: Deleting consensus metadata
I20260808 04:04:52.392640 16087 catalog_manager.cc:4841] TS 442fa005e54743748ca0af7671c4a79a (127.5.83.2:39807): tablet 6de7c77b84e6426aa8369ae8666af84f (table test-table [id=9f8f24289b304d96b7dac8f6bfb03421]) successfully deleted
I20260808 04:04:52.395138 16936 ts_tablet_manager.cc:1918] T 6de7c77b84e6426aa8369ae8666af84f P c230911ad282421cabd56a069e80976f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 04:04:52.395311 16936 log.cc:1196] T 6de7c77b84e6426aa8369ae8666af84f P c230911ad282421cabd56a069e80976f: Deleting WAL directory at /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161826605655-5452-0/minicluster-data/ts-2/wal/wals/6de7c77b84e6426aa8369ae8666af84f
I20260808 04:04:52.395776 16936 ts_tablet_manager.cc:1939] T 6de7c77b84e6426aa8369ae8666af84f P c230911ad282421cabd56a069e80976f: Deleting consensus metadata
I20260808 04:04:52.396768 16088 catalog_manager.cc:4841] TS c230911ad282421cabd56a069e80976f (127.5.83.3:43655): tablet 6de7c77b84e6426aa8369ae8666af84f (table test-table [id=9f8f24289b304d96b7dac8f6bfb03421]) successfully deleted
Aug 08 04:04:52 dist-test-slave-1l3l krb5kdc[16047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161891, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.83.1@KRBTEST.COM
I20260808 04:04:52.409354 17000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:52.386923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33310 (local address 127.5.83.1:35837)
0808 04:04:52.387078 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:52.387084 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:52.393316 (+ 6232us) server_negotiation.cc:407] Connection header received
0808 04:04:52.393584 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:52.393590 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:52.393670 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:52.393768 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:52.395244 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:52.396333 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:52.398090 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:52.398495 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:52.398586 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:52.403141 (+ 4555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:52.403159 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:52.403178 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:52.403225 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:52.406801 (+ 3576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:52.407394 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:52.407405 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:52.407408 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:52.407560 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:52.407909 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:52.407914 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:52.407917 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:52.408328 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:52.408625 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:52.408782 (+ 157us) server_negotiation.cc:299] Negotiation successful
0808 04:04:52.409015 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260808 04:04:52.410073 16429 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:04:52.507387 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 16224
I20260808 04:04:52.527788 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 16443
Aug 08 04:04:52 dist-test-slave-1l3l krb5kdc[16047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161891, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.83.2@KRBTEST.COM
I20260808 04:04:52.544780 16890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:52.528804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34716 (local address 127.5.83.2:39807)
0808 04:04:52.529053 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:52.529060 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:52.529083 (+ 23us) server_negotiation.cc:407] Connection header received
0808 04:04:52.529151 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:52.529156 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:52.529236 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:52.529350 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:52.530504 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:52.531633 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:52.533099 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:52.533786 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:52.533939 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:52.538204 (+ 4265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:52.538220 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:52.538240 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:52.538281 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:52.542032 (+ 3751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:52.542670 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:52.542679 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:52.542683 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:52.542864 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:52.543188 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:52.543193 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:52.543196 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:52.543642 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:52.544028 (+ 386us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:52.544195 (+ 167us) server_negotiation.cc:299] Negotiation successful
0808 04:04:52.544428 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260808 04:04:52.545535 16642 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:04:52.690435 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 16443
I20260808 04:04:52.716830 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 16652
Aug 08 04:04:52 dist-test-slave-1l3l krb5kdc[16047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161891, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.83.3@KRBTEST.COM
I20260808 04:04:52.730880 17041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:52.717583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41418 (local address 127.5.83.3:43655)
0808 04:04:52.718010 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:52.718014 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:52.718031 (+ 17us) server_negotiation.cc:407] Connection header received
0808 04:04:52.718084 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:52.718087 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:52.718156 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:52.718243 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:52.719445 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:52.720182 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:52.721377 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:52.721760 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:52.721846 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:52.725820 (+ 3974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:52.725835 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:52.725852 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:52.725885 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:52.728631 (+ 2746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:52.729218 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:52.729225 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:52.729227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:52.729357 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:52.729628 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:52.729631 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:52.729633 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:52.729962 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:52.730165 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:52.730373 (+ 208us) server_negotiation.cc:299] Negotiation successful
0808 04:04:52.730543 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":229,"threads_started":1}
I20260808 04:04:52.731523 16856 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:04:52.851995 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 16652
I20260808 04:04:52.880923 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 16061
I20260808 04:04:52.882050 16173 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:04:52.995832 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 16061
2026-08-08T04:04:53Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5462 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786161826605655-5452-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:04:53 dist-test-slave-1l3l krb5kdc[17047](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:04:53 dist-test-slave-1l3l krb5kdc[17047](info): set up 2 sockets
Aug 08 04:04:53 dist-test-slave-1l3l krb5kdc[17047](info): commencing operation
krb5kdc: starting...
W20260808 04:04:54.172456 5452 mini_kdc.cc:121] Time spent starting KDC: real 1.129s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786161826605655-5452-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786161826605655-5452-0/krb5kdc/test-user.keytab.
Aug 08 04:04:54 dist-test-slave-1l3l krb5kdc[17047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161894, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:04:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:04:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.83.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab.
Entry for principal kudu/127.5.83.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab.
I20260808 04:04:54.384722 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786161826605655-5452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786161826605655-5452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786161826605655-5452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:46237
--webserver_interface=127.5.83.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab
--principal=kudu/127.5.83.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:38273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.62:46237
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:04:54.699306 17066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:04:54.699599 17066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:04:54.699761 17066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:04:54.706720 17066 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:04:54.706827 17066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:04:54.706915 17066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:04:54.706992 17066 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:04:54.707058 17066 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:04:54.716572 17066 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:38273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786161826605655-5452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786161826605655-5452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.62:46237
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:46237
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786161826605655-5452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:04:54.717806 17066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:04:54.719316 17066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:04:54.730262 17081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:54.730327 17079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:54.730362 17077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:04:54.730940 17066 server_base.cc:1034] running on GCE node
I20260808 04:04:54.731666 17066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:04:54.732987 17066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:04:54.734180 17066 hybrid_clock.cc:648] HybridClock initialized: now 1786161894734133 us; error 36 us; skew 500 ppm
I20260808 04:04:54.734491 17066 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-08T04:04:54Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1801 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:04:54 dist-test-slave-1l3l krb5kdc[17088](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:04:54 dist-test-slave-1l3l krb5kdc[17088](info): set up 2 sockets
Aug 08 04:04:54 dist-test-slave-1l3l krb5kdc[17088](info): commencing operation
krb5kdc: starting...
W20260808 04:04:55.995625 5452 mini_kdc.cc:121] Time spent starting KDC: real 1.151s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/krb5kdc/test-user.keytab.
Aug 08 04:04:56 dist-test-slave-1l3l krb5kdc[17088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161896, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:04:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:04:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.83.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab.
Entry for principal kudu/127.5.83.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab.
I20260808 04:04:56.223522 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:40465
--webserver_interface=127.5.83.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab
--principal=kudu/127.5.83.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:41297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.62:40465
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:04:56.549335 17108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:04:56.549599 17108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:04:56.549693 17108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:04:56.556734 17108 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:04:56.556828 17108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:04:56.556883 17108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:04:56.556937 17108 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:04:56.556980 17108 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:04:56.566502 17108 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:41297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.62:40465
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab
--principal=kudu/127.5.83.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:40465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:04:56.567685 17108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:04:56.568918 17108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:04:56.579850 17114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:56.579895 17119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:56.579936 17116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:04:56.580992 17108 server_base.cc:1034] running on GCE node
I20260808 04:04:56.581655 17108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:04:56.582876 17108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:04:56.584066 17108 hybrid_clock.cc:648] HybridClock initialized: now 1786161896584030 us; error 25 us; skew 500 ppm
I20260808 04:04:56.584365 17108 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:04:56 dist-test-slave-1l3l krb5kdc[17088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161896, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:04:56.587146 17108 init.cc:377] Logged in from keytab as kudu/127.5.83.62@KRBTEST.COM (short username kudu)
I20260808 04:04:56.588513 17108 webserver.cc:460] Webserver started at http://127.5.83.62:38685/ using document root <none> and password file <none>
I20260808 04:04:56.589097 17108 fs_manager.cc:359] Metadata directory not provided
I20260808 04:04:56.589193 17108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:04:56.589450 17108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:04:56.592317 17108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/master-0/data/instance:
uuid: "fb429987cf064a1aab6e7c70021cf24a"
format_stamp: "Formatted at 2026-08-08 04:04:56 on dist-test-slave-1l3l"
server_key: "3e7729896298b689ea430e758affcfbc"
server_key_iv: "9d6b55e87442855b88832e14efcb46bd"
server_key_version: "encryptionkey@0"
I20260808 04:04:56.593031 17108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/master-0/wal/instance:
uuid: "fb429987cf064a1aab6e7c70021cf24a"
format_stamp: "Formatted at 2026-08-08 04:04:56 on dist-test-slave-1l3l"
server_key: "3e7729896298b689ea430e758affcfbc"
server_key_iv: "9d6b55e87442855b88832e14efcb46bd"
server_key_version: "encryptionkey@0"
I20260808 04:04:56.598199 17108 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260808 04:04:56.602535 17130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:56.604521 17108 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260808 04:04:56.604773 17108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/master-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/master-0/wal
uuid: "fb429987cf064a1aab6e7c70021cf24a"
format_stamp: "Formatted at 2026-08-08 04:04:56 on dist-test-slave-1l3l"
server_key: "3e7729896298b689ea430e758affcfbc"
server_key_iv: "9d6b55e87442855b88832e14efcb46bd"
server_key_version: "encryptionkey@0"
I20260808 04:04:56.605053 17108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:04:56.627974 17108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:04:56.631811 17108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:04:56.632071 17108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:04:56.649176 17108 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.62:40465
I20260808 04:04:56.649210 17210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.62:40465 every 8 connection(s)
I20260808 04:04:56.650486 17108 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
I20260808 04:04:56.652801 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 17108
I20260808 04:04:56.653017 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/master-0/wal/instance
I20260808 04:04:56.653855 5452 external_mini_cluster.cc:1428] Setting key 145d03a348b29ca3c069245fa0d5e596
I20260808 04:04:56.654641 17212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 08 04:04:56 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161896, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:04:56.668650 17212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb429987cf064a1aab6e7c70021cf24a: Bootstrap starting.
I20260808 04:04:56.671641 17212 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb429987cf064a1aab6e7c70021cf24a: Neither blocks nor log segments found. Creating new log.
I20260808 04:04:56.672817 17212 log.cc:824] T 00000000000000000000000000000000 P fb429987cf064a1aab6e7c70021cf24a: Log is configured to *not* fsync() on all Append() calls
I20260808 04:04:56.675849 17212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb429987cf064a1aab6e7c70021cf24a: No bootstrap required, opened a new log
I20260808 04:04:56.676254 17217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:56.655933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46384 (local address 127.5.83.62:40465)
0808 04:04:56.656633 (+ 700us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:56.656644 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:56.656683 (+ 39us) server_negotiation.cc:407] Connection header received
0808 04:04:56.658141 (+ 1458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:56.658161 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:56.658518 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:56.658846 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:56.660654 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:56.662022 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:56.663757 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:56.664405 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:56.664541 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:56.669156 (+ 4615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:56.669163 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:56.669175 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:56.669212 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:56.672119 (+ 2907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:56.672895 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:56.672901 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:56.672907 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:56.673080 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:56.673500 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:56.673506 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:56.673508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:56.674197 (+ 689us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:56.674408 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:56.674860 (+ 452us) server_negotiation.cc:299] Negotiation successful
0808 04:04:56.675186 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":318,"threads_started":1}
I20260808 04:04:56.682817 17212 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb429987cf064a1aab6e7c70021cf24a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb429987cf064a1aab6e7c70021cf24a" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 40465 } }
I20260808 04:04:56.683135 17212 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb429987cf064a1aab6e7c70021cf24a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:04:56.683228 17212 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb429987cf064a1aab6e7c70021cf24a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb429987cf064a1aab6e7c70021cf24a, State: Initialized, Role: FOLLOWER
I20260808 04:04:56.683728 17212 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb429987cf064a1aab6e7c70021cf24a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb429987cf064a1aab6e7c70021cf24a" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 40465 } }
I20260808 04:04:56.683969 17212 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb429987cf064a1aab6e7c70021cf24a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:04:56.684090 17212 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb429987cf064a1aab6e7c70021cf24a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:04:56.684240 17212 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb429987cf064a1aab6e7c70021cf24a [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:04:56.686172 17212 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb429987cf064a1aab6e7c70021cf24a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb429987cf064a1aab6e7c70021cf24a" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 40465 } }
I20260808 04:04:56.686606 17212 leader_election.cc:302] T 00000000000000000000000000000000 P fb429987cf064a1aab6e7c70021cf24a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb429987cf064a1aab6e7c70021cf24a; no voters:
I20260808 04:04:56.687145 17212 leader_election.cc:288] T 00000000000000000000000000000000 P fb429987cf064a1aab6e7c70021cf24a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:04:56.687310 17222 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb429987cf064a1aab6e7c70021cf24a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:04:56.688267 17222 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb429987cf064a1aab6e7c70021cf24a [term 1 LEADER]: Becoming Leader. State: Replica: fb429987cf064a1aab6e7c70021cf24a, State: Running, Role: LEADER
I20260808 04:04:56.688755 17222 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb429987cf064a1aab6e7c70021cf24a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb429987cf064a1aab6e7c70021cf24a" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 40465 } }
I20260808 04:04:56.689114 17212 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb429987cf064a1aab6e7c70021cf24a [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:04:56.691239 17224 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb429987cf064a1aab6e7c70021cf24a [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb429987cf064a1aab6e7c70021cf24a. Latest consensus state: current_term: 1 leader_uuid: "fb429987cf064a1aab6e7c70021cf24a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb429987cf064a1aab6e7c70021cf24a" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 40465 } } }
I20260808 04:04:56.691645 17224 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb429987cf064a1aab6e7c70021cf24a [sys.catalog]: This master's current role is: LEADER
I20260808 04:04:56.691869 17223 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb429987cf064a1aab6e7c70021cf24a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fb429987cf064a1aab6e7c70021cf24a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb429987cf064a1aab6e7c70021cf24a" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 40465 } } }
I20260808 04:04:56.692457 17223 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb429987cf064a1aab6e7c70021cf24a [sys.catalog]: This master's current role is: LEADER
I20260808 04:04:56.693600 17240 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:04:56.697966 17240 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:04:56.705276 17240 catalog_manager.cc:1347] Generated new cluster ID: d956e41e0fe64178952b7afc444c73c2
I20260808 04:04:56.705389 17240 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:04:56.725574 17240 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:04:56.726778 17240 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:04:56.747939 17240 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb429987cf064a1aab6e7c70021cf24a: Generated new TSK 0
I20260808 04:04:56.748944 17240 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.83.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.1.keytab.
Entry for principal kudu/127.5.83.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.1.keytab.
I20260808 04:04:56.807919 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.1:0
--local_ip_for_outbound_sockets=127.5.83.1
--webserver_interface=127.5.83.1
--webserver_port=0
--tserver_master_addrs=127.5.83.62:40465
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.1.keytab
--principal=kudu/127.5.83.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:41297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:04:57.128405 17255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:04:57.128671 17255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:04:57.128813 17255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:04:57.135912 17255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:04:57.136023 17255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:04:57.136171 17255 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.1
I20260808 04:04:57.145560 17255 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:41297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.1.keytab
--principal=kudu/127.5.83.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.83.1
--webserver_port=0
--tserver_master_addrs=127.5.83.62:40465
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.1
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:04:57.146746 17255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:04:57.148028 17255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:04:57.160192 17264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:57.160269 17267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:57.160243 17263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:04:57.161171 17255 server_base.cc:1034] running on GCE node
I20260808 04:04:57.161726 17255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:04:57.162547 17255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:04:57.163753 17255 hybrid_clock.cc:648] HybridClock initialized: now 1786161897163681 us; error 40 us; skew 500 ppm
I20260808 04:04:57.164044 17255 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:04:57 dist-test-slave-1l3l krb5kdc[17088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161897, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:04:57.167016 17255 init.cc:377] Logged in from keytab as kudu/127.5.83.1@KRBTEST.COM (short username kudu)
I20260808 04:04:57.168318 17255 webserver.cc:460] Webserver started at http://127.5.83.1:42505/ using document root <none> and password file <none>
I20260808 04:04:57.168895 17255 fs_manager.cc:359] Metadata directory not provided
I20260808 04:04:57.169003 17255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:04:57.169255 17255 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:04:57.172012 17255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-0/data/instance:
uuid: "2e62849b527641afa64507b4843bffc0"
format_stamp: "Formatted at 2026-08-08 04:04:57 on dist-test-slave-1l3l"
server_key: "d993e88ce68d4c794c3299afbc5eab18"
server_key_iv: "56a4f6a4ed1a9d3177ae46019419bcae"
server_key_version: "encryptionkey@0"
I20260808 04:04:57.172734 17255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-0/wal/instance:
uuid: "2e62849b527641afa64507b4843bffc0"
format_stamp: "Formatted at 2026-08-08 04:04:57 on dist-test-slave-1l3l"
server_key: "d993e88ce68d4c794c3299afbc5eab18"
server_key_iv: "56a4f6a4ed1a9d3177ae46019419bcae"
server_key_version: "encryptionkey@0"
I20260808 04:04:57.177451 17255 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 04:04:57.180668 17280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:57.181806 17255 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:04:57.181934 17255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-0/wal
uuid: "2e62849b527641afa64507b4843bffc0"
format_stamp: "Formatted at 2026-08-08 04:04:57 on dist-test-slave-1l3l"
server_key: "d993e88ce68d4c794c3299afbc5eab18"
server_key_iv: "56a4f6a4ed1a9d3177ae46019419bcae"
server_key_version: "encryptionkey@0"
I20260808 04:04:57.182092 17255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:04:57.194443 17255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:04:57.198092 17255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:04:57.198372 17255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:04:57.199241 17255 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:04:57.200570 17255 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:04:57.200651 17255 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:57.200752 17255 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:04:57.200798 17255 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:57.231096 17255 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.1:36033
I20260808 04:04:57.231139 17461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.1:36033 every 8 connection(s)
I20260808 04:04:57.232391 17255 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
I20260808 04:04:57.239675 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 17255
I20260808 04:04:57.239899 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-0/wal/instance
I20260808 04:04:57.240718 5452 external_mini_cluster.cc:1428] Setting key f3b9c2a6cca766536618b38596748132
Aug 08 04:04:57 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161897, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.1@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:04:57.252907 17465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:57.234901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.1:56711 (local address 127.5.83.62:40465)
0808 04:04:57.235348 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:57.235355 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:57.236317 (+ 962us) server_negotiation.cc:407] Connection header received
0808 04:04:57.238044 (+ 1727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:57.238048 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:57.238114 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:57.238206 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:57.239658 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:57.240755 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:57.242780 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:57.243363 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:57.243494 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:57.247003 (+ 3509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:57.247009 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:57.247013 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:57.247060 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:57.250004 (+ 2944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:57.250689 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:57.250694 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:57.250697 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:57.250834 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:57.251263 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:57.251266 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:57.251269 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:57.251592 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:57.251739 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:57.252356 (+ 617us) server_negotiation.cc:299] Negotiation successful
0808 04:04:57.252566 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":190,"threads_started":1}
I20260808 04:04:57.253679 17467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:57.235445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:40465 (local address 127.5.83.1:56711)
0808 04:04:57.236170 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:57.236209 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:57.237806 (+ 1597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:57.238435 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:57.238444 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:57.238929 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:04:57.239486 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:57.239503 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:57.240935 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:57.240941 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:57.242635 (+ 1694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:57.242651 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:57.243560 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:57.243565 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:57.243831 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:57.244489 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:04:57.244511 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:04:57.246757 (+ 2246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:04:57.250142 (+ 3385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:57.250149 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:57.250161 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:57.250572 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:57.250968 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:57.250971 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:57.250974 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:57.251159 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:57.251699 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:04:57.251723 (+ 24us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:04:57.252037 (+ 314us) client_negotiation.cc:769] Sending connection context
0808 04:04:57.252255 (+ 218us) client_negotiation.cc:240] Negotiation successful
0808 04:04:57.252628 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":305,"threads_started":1}
I20260808 04:04:57.255155 17462 heartbeater.cc:347] Connected to a master server at 127.5.83.62:40465
I20260808 04:04:57.255446 17462 heartbeater.cc:464] Registering TS with master...
I20260808 04:04:57.256166 17462 heartbeater.cc:511] Master 127.5.83.62:40465 requested a full tablet report, sending...
I20260808 04:04:57.258388 17159 ts_manager.cc:194] Registered new tserver with Master: 2e62849b527641afa64507b4843bffc0 (127.5.83.1:36033)
I20260808 04:04:57.260255 17159 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.1@KRBTEST.COM'} at 127.5.83.1:56711
WARNING: no policy specified for kudu/127.5.83.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.2.keytab.
Entry for principal kudu/127.5.83.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.2.keytab.
I20260808 04:04:57.294950 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.2:0
--local_ip_for_outbound_sockets=127.5.83.2
--webserver_interface=127.5.83.2
--webserver_port=0
--tserver_master_addrs=127.5.83.62:40465
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.2.keytab
--principal=kudu/127.5.83.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:41297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:04:57.612742 17470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:04:57.613011 17470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:04:57.613152 17470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:04:57.620196 17470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:04:57.620321 17470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:04:57.620481 17470 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.2
I20260808 04:04:57.629806 17470 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:41297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.2.keytab
--principal=kudu/127.5.83.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.83.2
--webserver_port=0
--tserver_master_addrs=127.5.83.62:40465
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.2
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:04:57.631026 17470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:04:57.632335 17470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:04:57.644860 17479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:57.644785 17484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:57.644817 17482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:04:57.645638 17470 server_base.cc:1034] running on GCE node
I20260808 04:04:57.646166 17470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:04:57.647027 17470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:04:57.648222 17470 hybrid_clock.cc:648] HybridClock initialized: now 1786161897648177 us; error 37 us; skew 500 ppm
I20260808 04:04:57.648512 17470 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:04:57 dist-test-slave-1l3l krb5kdc[17088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161897, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:04:57.651563 17470 init.cc:377] Logged in from keytab as kudu/127.5.83.2@KRBTEST.COM (short username kudu)
I20260808 04:04:57.652983 17470 webserver.cc:460] Webserver started at http://127.5.83.2:38211/ using document root <none> and password file <none>
I20260808 04:04:57.653625 17470 fs_manager.cc:359] Metadata directory not provided
I20260808 04:04:57.653745 17470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:04:57.654048 17470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:04:57.656903 17470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-1/data/instance:
uuid: "d4746fb9b7b24e60b7306eced48511b5"
format_stamp: "Formatted at 2026-08-08 04:04:57 on dist-test-slave-1l3l"
server_key: "e329cd10cfa4bb5e9a1826018cd827f6"
server_key_iv: "eb5241d39bd8ba0d4ed2160d24426f74"
server_key_version: "encryptionkey@0"
I20260808 04:04:57.657644 17470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-1/wal/instance:
uuid: "d4746fb9b7b24e60b7306eced48511b5"
format_stamp: "Formatted at 2026-08-08 04:04:57 on dist-test-slave-1l3l"
server_key: "e329cd10cfa4bb5e9a1826018cd827f6"
server_key_iv: "eb5241d39bd8ba0d4ed2160d24426f74"
server_key_version: "encryptionkey@0"
I20260808 04:04:57.662664 17470 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260808 04:04:57.665977 17493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:57.667193 17470 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260808 04:04:57.667336 17470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-1/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-1/wal
uuid: "d4746fb9b7b24e60b7306eced48511b5"
format_stamp: "Formatted at 2026-08-08 04:04:57 on dist-test-slave-1l3l"
server_key: "e329cd10cfa4bb5e9a1826018cd827f6"
server_key_iv: "eb5241d39bd8ba0d4ed2160d24426f74"
server_key_version: "encryptionkey@0"
I20260808 04:04:57.667508 17470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:04:57.687131 17470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:04:57.691087 17470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:04:57.691339 17470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:04:57.692237 17470 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:04:57.693651 17470 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:04:57.693725 17470 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:57.693817 17470 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:04:57.693867 17470 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:57.723955 17470 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.2:37893
I20260808 04:04:57.724025 17657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.2:37893 every 8 connection(s)
I20260808 04:04:57.725181 17470 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
I20260808 04:04:57.726275 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 17470
I20260808 04:04:57.726475 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-1/wal/instance
I20260808 04:04:57.727087 5452 external_mini_cluster.cc:1428] Setting key c903e73ae58e9174b0320c2ba6f20ddc
Aug 08 04:04:57 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161897, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.2@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:04:57.747135 17465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:57.727810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.2:55087 (local address 127.5.83.62:40465)
0808 04:04:57.728001 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:57.728006 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:57.729351 (+ 1345us) server_negotiation.cc:407] Connection header received
0808 04:04:57.731833 (+ 2482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:57.731837 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:57.731930 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:57.732018 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:57.733823 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:57.734734 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:57.736780 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:57.737274 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:57.737375 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:57.741188 (+ 3813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:57.741194 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:57.741198 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:57.741234 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:57.744226 (+ 2992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:57.744940 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:57.744945 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:57.744948 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:57.745095 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:57.745507 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:57.745512 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:57.745515 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:57.745829 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:57.745971 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:57.746609 (+ 638us) server_negotiation.cc:299] Negotiation successful
0808 04:04:57.746818 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 04:04:57.747857 17661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:57.728388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:40465 (local address 127.5.83.2:55087)
0808 04:04:57.729200 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:57.729249 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:57.731514 (+ 2265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:57.732261 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:57.732272 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:57.732946 (+ 674us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:04:57.733647 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:57.733670 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:57.734904 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:57.734909 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:57.736657 (+ 1748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:57.736673 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:57.737496 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:57.737503 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:57.737758 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:57.738443 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:04:57.738475 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:04:57.740992 (+ 2517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:04:57.744348 (+ 3356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:57.744354 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:57.744365 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:57.744812 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:57.745196 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:57.745199 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:57.745201 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:57.745370 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:57.745947 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:04:57.745952 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:04:57.746229 (+ 277us) client_negotiation.cc:769] Sending connection context
0808 04:04:57.746462 (+ 233us) client_negotiation.cc:240] Negotiation successful
0808 04:04:57.746806 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":315,"threads_started":1}
I20260808 04:04:57.749591 17659 heartbeater.cc:347] Connected to a master server at 127.5.83.62:40465
I20260808 04:04:57.749861 17659 heartbeater.cc:464] Registering TS with master...
I20260808 04:04:57.750535 17659 heartbeater.cc:511] Master 127.5.83.62:40465 requested a full tablet report, sending...
I20260808 04:04:57.751948 17159 ts_manager.cc:194] Registered new tserver with Master: d4746fb9b7b24e60b7306eced48511b5 (127.5.83.2:37893)
I20260808 04:04:57.752688 17159 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.2@KRBTEST.COM'} at 127.5.83.2:55087
WARNING: no policy specified for kudu/127.5.83.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.3.keytab.
Entry for principal kudu/127.5.83.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.3.keytab.
I20260808 04:04:57.782196 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.3:0
--local_ip_for_outbound_sockets=127.5.83.3
--webserver_interface=127.5.83.3
--webserver_port=0
--tserver_master_addrs=127.5.83.62:40465
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.3.keytab
--principal=kudu/127.5.83.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:41297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:04:58.099666 17666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:04:58.099946 17666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:04:58.100044 17666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:04:58.106895 17666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:04:58.106989 17666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:04:58.107100 17666 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.3
I20260808 04:04:58.116348 17666 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:41297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.3.keytab
--principal=kudu/127.5.83.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.83.3
--webserver_port=0
--tserver_master_addrs=127.5.83.62:40465
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.3
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:04:58.117499 17666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:04:58.118744 17666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:04:58.131410 17674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:58.131477 17675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:58.131368 17677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:04:58.132319 17666 server_base.cc:1034] running on GCE node
I20260808 04:04:58.132843 17666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:04:58.133713 17666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:04:58.134907 17666 hybrid_clock.cc:648] HybridClock initialized: now 1786161898134862 us; error 31 us; skew 500 ppm
I20260808 04:04:58.135161 17666 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:04:58 dist-test-slave-1l3l krb5kdc[17088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161898, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:04:58.138129 17666 init.cc:377] Logged in from keytab as kudu/127.5.83.3@KRBTEST.COM (short username kudu)
I20260808 04:04:58.139401 17666 webserver.cc:460] Webserver started at http://127.5.83.3:37063/ using document root <none> and password file <none>
I20260808 04:04:58.140043 17666 fs_manager.cc:359] Metadata directory not provided
I20260808 04:04:58.140141 17666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:04:58.140467 17666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:04:58.143226 17666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-2/data/instance:
uuid: "4c293b7b37b14b82842c337daf292a69"
format_stamp: "Formatted at 2026-08-08 04:04:58 on dist-test-slave-1l3l"
server_key: "7c99f00aeb7327e504ef0c346eaae098"
server_key_iv: "fa0d3461f6f741ea08e3ee34a531be25"
server_key_version: "encryptionkey@0"
I20260808 04:04:58.144009 17666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-2/wal/instance:
uuid: "4c293b7b37b14b82842c337daf292a69"
format_stamp: "Formatted at 2026-08-08 04:04:58 on dist-test-slave-1l3l"
server_key: "7c99f00aeb7327e504ef0c346eaae098"
server_key_iv: "fa0d3461f6f741ea08e3ee34a531be25"
server_key_version: "encryptionkey@0"
I20260808 04:04:58.148684 17666 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260808 04:04:58.151794 17691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:58.153085 17666 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 04:04:58.153227 17666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-2/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-2/wal
uuid: "4c293b7b37b14b82842c337daf292a69"
format_stamp: "Formatted at 2026-08-08 04:04:58 on dist-test-slave-1l3l"
server_key: "7c99f00aeb7327e504ef0c346eaae098"
server_key_iv: "fa0d3461f6f741ea08e3ee34a531be25"
server_key_version: "encryptionkey@0"
I20260808 04:04:58.153398 17666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:04:58.188552 17666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:04:58.192781 17666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:04:58.193037 17666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:04:58.193964 17666 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:04:58.195230 17666 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:04:58.195326 17666 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:58.195464 17666 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:04:58.195519 17666 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:58.225481 17666 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.3:35613
I20260808 04:04:58.225515 17852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.3:35613 every 8 connection(s)
I20260808 04:04:58.226768 17666 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
I20260808 04:04:58.233783 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 17666
I20260808 04:04:58.233956 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-2/wal/instance
I20260808 04:04:58.234477 5452 external_mini_cluster.cc:1428] Setting key 56b3da20c1590dcf2ec5261e4480cab2
Aug 08 04:04:58 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161898, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.3@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:04:58.246026 17465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:58.229230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.3:37565 (local address 127.5.83.62:40465)
0808 04:04:58.229419 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:58.229423 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:58.230583 (+ 1160us) server_negotiation.cc:407] Connection header received
0808 04:04:58.232411 (+ 1828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:58.232417 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:58.232497 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:58.232587 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:58.234039 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:58.234877 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:58.236759 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:58.237182 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:58.237310 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:58.240585 (+ 3275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:58.240589 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:58.240593 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:58.240624 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:58.243134 (+ 2510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:58.243840 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:58.243845 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:58.243848 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:58.243966 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:58.244385 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:58.244389 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:58.244391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:58.244652 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:58.244785 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:58.245456 (+ 671us) server_negotiation.cc:299] Negotiation successful
0808 04:04:58.245684 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 04:04:58.246851 17858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:58.229728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:40465 (local address 127.5.83.3:37565)
0808 04:04:58.230426 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:58.230469 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:58.232152 (+ 1683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:58.232820 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:58.232828 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:58.233303 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:04:58.233822 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:58.233839 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:58.235045 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:58.235050 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:58.236628 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:58.236645 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:58.237423 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:58.237431 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:58.237660 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:58.238218 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:04:58.238244 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:04:58.240384 (+ 2140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:04:58.243266 (+ 2882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:58.243274 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:58.243288 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:58.243683 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:58.244072 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:04:58.244077 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:04:58.244081 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:04:58.244260 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:04:58.244777 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:04:58.244785 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:04:58.245093 (+ 308us) client_negotiation.cc:769] Sending connection context
0808 04:04:58.245331 (+ 238us) client_negotiation.cc:240] Negotiation successful
0808 04:04:58.245801 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":274,"threads_started":1}
I20260808 04:04:58.248227 17856 heartbeater.cc:347] Connected to a master server at 127.5.83.62:40465
I20260808 04:04:58.248538 17856 heartbeater.cc:464] Registering TS with master...
I20260808 04:04:58.249408 17856 heartbeater.cc:511] Master 127.5.83.62:40465 requested a full tablet report, sending...
I20260808 04:04:58.250841 17159 ts_manager.cc:194] Registered new tserver with Master: 4c293b7b37b14b82842c337daf292a69 (127.5.83.3:35613)
I20260808 04:04:58.251616 17159 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.3@KRBTEST.COM'} at 127.5.83.3:37565
I20260808 04:04:58.261721 5452 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:04:58.263015 17462 heartbeater.cc:503] Master 127.5.83.62:40465 was elected leader, sending a full tablet report...
WARNING: no policy specified for oryx/127.5.83.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.83.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.83.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/krb5kdc/oryx_127.5.83.1.keytab.
Entry for principal oryx/127.5.83.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/krb5kdc/oryx_127.5.83.1.keytab.
WARNING: no policy specified for oryx/127.5.83.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.83.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.83.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/krb5kdc/oryx_127.5.83.2.keytab.
Entry for principal oryx/127.5.83.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/krb5kdc/oryx_127.5.83.2.keytab.
WARNING: no policy specified for oryx/127.5.83.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.83.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.83.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/krb5kdc/oryx_127.5.83.3.keytab.
Entry for principal oryx/127.5.83.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/krb5kdc/oryx_127.5.83.3.keytab.
I20260808 04:04:58.396657 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 17255
Aug 08 04:04:58 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161896, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.1@KRBTEST.COM
I20260808 04:04:58.411653 17873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:58.397731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34828 (local address 127.5.83.1:36033)
0808 04:04:58.398359 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:58.398366 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:58.398396 (+ 30us) server_negotiation.cc:407] Connection header received
0808 04:04:58.398458 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:58.398465 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:58.398672 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:58.398812 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:58.400145 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:58.401196 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:58.402348 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:58.402904 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:58.402998 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:58.406629 (+ 3631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:58.406641 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:58.406656 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:58.406691 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:58.409426 (+ 2735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:58.410056 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:58.410063 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:58.410065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:58.410195 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:58.410470 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:58.410473 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:58.410475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:58.410785 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:58.411013 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:58.411190 (+ 177us) server_negotiation.cc:299] Negotiation successful
0808 04:04:58.411374 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":320,"threads_started":1}
I20260808 04:04:58.412839 17453 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:04:58.494841 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 17255
I20260808 04:04:58.514577 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 17470
Aug 08 04:04:58 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161896, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.2@KRBTEST.COM
I20260808 04:04:58.529124 17875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:58.515445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33284 (local address 127.5.83.2:37893)
0808 04:04:58.515972 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:58.515979 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:58.516004 (+ 25us) server_negotiation.cc:407] Connection header received
0808 04:04:58.516065 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:58.516071 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:58.516269 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:58.516406 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:58.517500 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:58.518488 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:58.519628 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:58.520309 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:58.520424 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:58.524170 (+ 3746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:58.524181 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:58.524193 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:58.524224 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:58.526897 (+ 2673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:58.527451 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:58.527457 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:58.527459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:58.527570 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:58.527934 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:58.527937 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:58.527939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:58.528256 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:58.528495 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:58.528670 (+ 175us) server_negotiation.cc:299] Negotiation successful
0808 04:04:58.528847 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":268,"threads_started":1}
I20260808 04:04:58.530289 17652 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:04:58.610528 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 17470
I20260808 04:04:58.629916 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 17666
Aug 08 04:04:58 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161896, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.3@KRBTEST.COM
I20260808 04:04:58.645510 17878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:58.630980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35554 (local address 127.5.83.3:35613)
0808 04:04:58.631646 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:58.631652 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:58.631682 (+ 30us) server_negotiation.cc:407] Connection header received
0808 04:04:58.631810 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:58.631820 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:58.632099 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:58.632231 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:58.633506 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:58.634509 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:58.635694 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:58.636276 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:58.636371 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:58.640508 (+ 4137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:04:58.640519 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:04:58.640532 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:04:58.640567 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:04:58.643235 (+ 2668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:58.643824 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:58.643831 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:58.643833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:58.643990 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:04:58.644252 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:04:58.644255 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:04:58.644257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:04:58.644562 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:04:58.644789 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0808 04:04:58.644986 (+ 197us) server_negotiation.cc:299] Negotiation successful
0808 04:04:58.645182 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":361,"threads_started":1}
I20260808 04:04:58.646677 17847 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:04:58.727155 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 17666
I20260808 04:04:58.748978 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 17108
I20260808 04:04:58.750105 17206 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:04:58.851778 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 17108
I20260808 04:04:58.872519 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:40465
--webserver_interface=127.5.83.62
--webserver_port=38685
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab
--principal=kudu/127.5.83.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:41297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.62:40465
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:04:59.195211 17882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:04:59.195499 17882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:04:59.195628 17882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:04:59.202587 17882 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:04:59.202682 17882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:04:59.202752 17882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:04:59.202827 17882 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:04:59.202889 17882 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:04:59.212275 17882 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:41297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.62:40465
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab
--principal=kudu/127.5.83.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:40465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.62
--webserver_port=38685
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:04:59.213460 17882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:04:59.214727 17882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:04:59.225775 17887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:59.225807 17888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:59.225733 17892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:04:59.226775 17882 server_base.cc:1034] running on GCE node
I20260808 04:04:59.227494 17882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:04:59.228840 17882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:04:59.230043 17882 hybrid_clock.cc:648] HybridClock initialized: now 1786161899229993 us; error 35 us; skew 500 ppm
I20260808 04:04:59.230346 17882 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:04:59 dist-test-slave-1l3l krb5kdc[17088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161899, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:04:59.233350 17882 init.cc:377] Logged in from keytab as kudu/127.5.83.62@KRBTEST.COM (short username kudu)
I20260808 04:04:59.234678 17882 webserver.cc:460] Webserver started at http://127.5.83.62:38685/ using document root <none> and password file <none>
I20260808 04:04:59.235260 17882 fs_manager.cc:359] Metadata directory not provided
I20260808 04:04:59.235395 17882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:04:59.240521 17882 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260808 04:04:59.243073 17902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:59.244421 17882 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 04:04:59.244561 17882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/master-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/master-0/wal
uuid: "fb429987cf064a1aab6e7c70021cf24a"
format_stamp: "Formatted at 2026-08-08 04:04:56 on dist-test-slave-1l3l"
server_key: "3e7729896298b689ea430e758affcfbc"
server_key_iv: "9d6b55e87442855b88832e14efcb46bd"
server_key_version: "encryptionkey@0"
I20260808 04:04:59.245451 17882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:04:59.262218 17882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:04:59.266317 17882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:04:59.266573 17882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:04:59.283010 17882 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.62:40465
I20260808 04:04:59.283046 17977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.62:40465 every 8 connection(s)
I20260808 04:04:59.284433 17882 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
I20260808 04:04:59.290966 17979 sys_catalog.cc:263] Verifying existing consensus state
I20260808 04:04:59.294030 17979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb429987cf064a1aab6e7c70021cf24a: Bootstrap starting.
I20260808 04:04:59.294528 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 17882
I20260808 04:04:59.295324 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.1:36033
--local_ip_for_outbound_sockets=127.5.83.1
--tserver_master_addrs=127.5.83.62:40465
--webserver_port=42505
--webserver_interface=127.5.83.1
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.1.keytab
--principal=kudu/127.5.83.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:41297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.5.83.1
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/krb5kdc/oryx_127.5.83.1.keytab with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:04:59.314924 17979 log.cc:824] T 00000000000000000000000000000000 P fb429987cf064a1aab6e7c70021cf24a: Log is configured to *not* fsync() on all Append() calls
I20260808 04:04:59.326478 17979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb429987cf064a1aab6e7c70021cf24a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 04:04:59.327097 17979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb429987cf064a1aab6e7c70021cf24a: Bootstrap complete.
I20260808 04:04:59.340411 17979 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb429987cf064a1aab6e7c70021cf24a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb429987cf064a1aab6e7c70021cf24a" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 40465 } }
I20260808 04:04:59.341517 17979 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb429987cf064a1aab6e7c70021cf24a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb429987cf064a1aab6e7c70021cf24a, State: Initialized, Role: FOLLOWER
I20260808 04:04:59.342206 17979 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb429987cf064a1aab6e7c70021cf24a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb429987cf064a1aab6e7c70021cf24a" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 40465 } }
I20260808 04:04:59.342476 17979 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb429987cf064a1aab6e7c70021cf24a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:04:59.342592 17979 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb429987cf064a1aab6e7c70021cf24a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:04:59.342784 17979 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb429987cf064a1aab6e7c70021cf24a [term 1 FOLLOWER]: Advancing to term 2
I20260808 04:04:59.348050 17979 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb429987cf064a1aab6e7c70021cf24a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb429987cf064a1aab6e7c70021cf24a" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 40465 } }
I20260808 04:04:59.348634 17979 leader_election.cc:302] T 00000000000000000000000000000000 P fb429987cf064a1aab6e7c70021cf24a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb429987cf064a1aab6e7c70021cf24a; no voters:
I20260808 04:04:59.349272 17979 leader_election.cc:288] T 00000000000000000000000000000000 P fb429987cf064a1aab6e7c70021cf24a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 04:04:59.349392 17985 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb429987cf064a1aab6e7c70021cf24a [term 2 FOLLOWER]: Leader election won for term 2
I20260808 04:04:59.350554 17985 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb429987cf064a1aab6e7c70021cf24a [term 2 LEADER]: Becoming Leader. State: Replica: fb429987cf064a1aab6e7c70021cf24a, State: Running, Role: LEADER
I20260808 04:04:59.351065 17985 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb429987cf064a1aab6e7c70021cf24a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb429987cf064a1aab6e7c70021cf24a" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 40465 } }
I20260808 04:04:59.351311 17979 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb429987cf064a1aab6e7c70021cf24a [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:04:59.353277 17987 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb429987cf064a1aab6e7c70021cf24a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "fb429987cf064a1aab6e7c70021cf24a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb429987cf064a1aab6e7c70021cf24a" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 40465 } } }
I20260808 04:04:59.353566 17987 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb429987cf064a1aab6e7c70021cf24a [sys.catalog]: This master's current role is: LEADER
I20260808 04:04:59.355008 17992 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:04:59.354893 17988 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb429987cf064a1aab6e7c70021cf24a [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb429987cf064a1aab6e7c70021cf24a. Latest consensus state: current_term: 2 leader_uuid: "fb429987cf064a1aab6e7c70021cf24a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb429987cf064a1aab6e7c70021cf24a" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 40465 } } }
I20260808 04:04:59.355223 17988 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb429987cf064a1aab6e7c70021cf24a [sys.catalog]: This master's current role is: LEADER
I20260808 04:04:59.360688 17992 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:04:59.362020 17992 catalog_manager.cc:1259] Loaded cluster ID: d956e41e0fe64178952b7afc444c73c2
I20260808 04:04:59.362159 17992 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:04:59.366040 17992 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:04:59.367456 17992 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fb429987cf064a1aab6e7c70021cf24a: Loaded TSK: 0
I20260808 04:04:59.368963 17992 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 04:04:59.638686 17983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:04:59.638952 17983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:04:59.639070 17983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:04:59.646319 17983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:04:59.646425 17983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:04:59.646562 17983 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.1
I20260808 04:04:59.656258 17983 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:41297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/krb5kdc/oryx_127.5.83.1.keytab
--principal=oryx/127.5.83.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.1:36033
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.83.1
--webserver_port=42505
--tserver_master_addrs=127.5.83.62:40465
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.1
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:04:59.657521 17983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:04:59.658807 17983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:04:59.672613 18019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:59.672595 18020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:04:59.673352 18022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:04:59.673990 17983 server_base.cc:1034] running on GCE node
I20260808 04:04:59.674538 17983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:04:59.675460 17983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:04:59.676669 17983 hybrid_clock.cc:648] HybridClock initialized: now 1786161899676617 us; error 47 us; skew 500 ppm
I20260808 04:04:59.676955 17983 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:04:59 dist-test-slave-1l3l krb5kdc[17088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161899, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.83.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:04:59.680142 17983 init.cc:377] Logged in from keytab as oryx/127.5.83.1@KRBTEST.COM (short username oryx)
I20260808 04:04:59.681602 17983 webserver.cc:460] Webserver started at http://127.5.83.1:42505/ using document root <none> and password file <none>
I20260808 04:04:59.682236 17983 fs_manager.cc:359] Metadata directory not provided
I20260808 04:04:59.682365 17983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:04:59.687433 17983 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260808 04:04:59.690021 18038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:59.691255 17983 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260808 04:04:59.691390 17983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-0/wal
uuid: "2e62849b527641afa64507b4843bffc0"
format_stamp: "Formatted at 2026-08-08 04:04:57 on dist-test-slave-1l3l"
server_key: "d993e88ce68d4c794c3299afbc5eab18"
server_key_iv: "56a4f6a4ed1a9d3177ae46019419bcae"
server_key_version: "encryptionkey@0"
I20260808 04:04:59.692354 17983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:04:59.705192 17983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:04:59.708847 17983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:04:59.709093 17983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:04:59.709978 17983 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:04:59.711234 17983 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:04:59.711329 17983 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:59.711438 17983 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:04:59.711516 17983 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:04:59.741701 17983 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.1:36033
I20260808 04:04:59.741771 18186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.1:36033 every 8 connection(s)
I20260808 04:04:59.743206 17983 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
I20260808 04:04:59.744491 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 17983
I20260808 04:04:59.745090 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.2:37893
--local_ip_for_outbound_sockets=127.5.83.2
--tserver_master_addrs=127.5.83.62:40465
--webserver_port=38211
--webserver_interface=127.5.83.2
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.2.keytab
--principal=kudu/127.5.83.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:41297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.5.83.2
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/krb5kdc/oryx_127.5.83.2.keytab with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 04:04:59 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.1@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:04:59 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.1@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
W20260808 04:04:59.762549 18194 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:04:59.746378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.1:44807 (local address 127.5.83.62:40465)
0808 04:04:59.747078 (+ 700us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:59.747094 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:59.747559 (+ 465us) server_negotiation.cc:407] Connection header received
0808 04:04:59.749760 (+ 2201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:59.749773 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:59.750081 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:59.750384 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:59.751927 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:59.753689 (+ 1762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:59.755460 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:59.756136 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:59.756261 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:59.761755 (+ 5494us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.1:44807: BlockingRecv error: recv got EOF from 127.5.83.1:44807 (error 108)
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":249,"threads_started":1}
I20260808 04:04:59.762573 18195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:59.746577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:40465 (local address 127.5.83.1:44807)
0808 04:04:59.747375 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:59.747425 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:59.749229 (+ 1804us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:59.750561 (+ 1332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:59.750570 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:59.751133 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:04:59.751685 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:59.751733 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:59.753838 (+ 2105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:59.753842 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:59.755313 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:59.755329 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:59.756352 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:59.756359 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:59.756622 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:59.757282 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:04:59.757309 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:04:59.761485 (+ 4176us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.62:40465: Server oryx/127.5.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":384,"threads_started":1}
W20260808 04:04:59.763195 18190 heartbeater.cc:644] Failed to heartbeat to 127.5.83.62:40465 (0 consecutive failures): Not authorized: Failed to ping master at 127.5.83.62:40465: Client connection negotiation failed: client connection to 127.5.83.62:40465: Server oryx/127.5.83.62@KRBTEST.COM not found in Kerberos database
Aug 08 04:04:59 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.1@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:04:59 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.1@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:04:59.771231 18195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:59.763748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:40465 (local address 127.5.83.1:56171)
0808 04:04:59.763934 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:59.763953 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:59.764045 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:59.764321 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:59.764323 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:59.764610 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:04:59.764737 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:59.764744 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:59.766007 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:59.766011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:59.766857 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:59.766867 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:59.767744 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:59.767752 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:59.767871 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:59.768385 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:04:59.768404 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:04:59.771040 (+ 2636us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.62:40465: Server oryx/127.5.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":52}
W20260808 04:04:59.771649 18194 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:04:59.763920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.1:56171 (local address 127.5.83.62:40465)
0808 04:04:59.764088 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:59.764093 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:59.764110 (+ 17us) server_negotiation.cc:407] Connection header received
0808 04:04:59.764153 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:59.764157 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:59.764215 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:59.764297 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:59.764874 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:59.765888 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:59.767003 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:59.767588 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:59.767737 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:59.771273 (+ 3536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.1:56171: BlockingRecv error: recv got EOF from 127.5.83.1:56171 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 08 04:04:59 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.1@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:04:59 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.1@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:04:59.778926 18195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:04:59.771905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:40465 (local address 127.5.83.1:39141)
0808 04:04:59.772056 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0808 04:04:59.772071 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:04:59.772142 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:04:59.772507 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:04:59.772512 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:04:59.772774 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:04:59.772905 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:59.772913 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:59.773817 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:59.773821 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:59.774687 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:04:59.774698 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:04:59.775352 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:04:59.775358 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:04:59.775511 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:59.776064 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:04:59.776080 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:04:59.778661 (+ 2581us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.62:40465: Server oryx/127.5.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":48}
W20260808 04:04:59.779121 18194 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:04:59.772119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.1:39141 (local address 127.5.83.62:40465)
0808 04:04:59.772276 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0808 04:04:59.772279 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:04:59.772292 (+ 13us) server_negotiation.cc:407] Connection header received
0808 04:04:59.772325 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:04:59.772328 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:04:59.772388 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:04:59.772467 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:04:59.773029 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:59.773686 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:59.774813 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:04:59.775210 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:04:59.775327 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:04:59.778862 (+ 3535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.1:39141: BlockingRecv error: recv got EOF from 127.5.83.1:39141 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260808 04:04:59.779201 18190 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 04:05:00.071081 18197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:05:00.071359 18197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:05:00.071501 18197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:05:00.078696 18197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:05:00.078822 18197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:05:00.078974 18197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.2
I20260808 04:05:00.088452 18197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:41297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/krb5kdc/oryx_127.5.83.2.keytab
--principal=oryx/127.5.83.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.2:37893
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.83.2
--webserver_port=38211
--tserver_master_addrs=127.5.83.62:40465
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.2
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:05:00.089807 18197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:05:00.091071 18197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:05:00.103163 18205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:05:00.103133 18206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:05:00.103209 18208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:05:00.104231 18197 server_base.cc:1034] running on GCE node
I20260808 04:05:00.104770 18197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:05:00.105614 18197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:05:00.106803 18197 hybrid_clock.cc:648] HybridClock initialized: now 1786161900106761 us; error 35 us; skew 500 ppm
I20260808 04:05:00.107079 18197 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:05:00 dist-test-slave-1l3l krb5kdc[17088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161900, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.83.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:05:00.109987 18197 init.cc:377] Logged in from keytab as oryx/127.5.83.2@KRBTEST.COM (short username oryx)
I20260808 04:05:00.111255 18197 webserver.cc:460] Webserver started at http://127.5.83.2:38211/ using document root <none> and password file <none>
I20260808 04:05:00.111919 18197 fs_manager.cc:359] Metadata directory not provided
I20260808 04:05:00.112046 18197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:05:00.117065 18197 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260808 04:05:00.119526 18219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:05:00.120836 18197 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260808 04:05:00.120987 18197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-1/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-1/wal
uuid: "d4746fb9b7b24e60b7306eced48511b5"
format_stamp: "Formatted at 2026-08-08 04:04:57 on dist-test-slave-1l3l"
server_key: "e329cd10cfa4bb5e9a1826018cd827f6"
server_key_iv: "eb5241d39bd8ba0d4ed2160d24426f74"
server_key_version: "encryptionkey@0"
I20260808 04:05:00.122001 18197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:05:00.137135 18197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:05:00.140861 18197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:05:00.141111 18197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:05:00.142035 18197 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:05:00.143342 18197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:05:00.143430 18197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:05:00.143534 18197 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:05:00.143603 18197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:05:00.172362 18197 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.2:37893
I20260808 04:05:00.172435 18379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.2:37893 every 8 connection(s)
I20260808 04:05:00.173728 18197 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
I20260808 04:05:00.183509 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 18197
I20260808 04:05:00.184329 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.3:35613
--local_ip_for_outbound_sockets=127.5.83.3
--tserver_master_addrs=127.5.83.62:40465
--webserver_port=37063
--webserver_interface=127.5.83.3
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.3.keytab
--principal=kudu/127.5.83.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:41297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.5.83.3
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/krb5kdc/oryx_127.5.83.3.keytab with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 04:05:00 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.2@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:05:00 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.2@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
W20260808 04:05:00.189170 18194 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:05:00.176171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.2:55805 (local address 127.5.83.62:40465)
0808 04:05:00.176386 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0808 04:05:00.176392 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:05:00.177550 (+ 1158us) server_negotiation.cc:407] Connection header received
0808 04:05:00.179287 (+ 1737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:05:00.179292 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:05:00.179364 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:05:00.179456 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:05:00.180869 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:00.181574 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:00.183001 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:00.183422 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:00.183515 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:00.188982 (+ 5467us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.2:55805: BlockingRecv error: recv got EOF from 127.5.83.2:55805 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 04:05:00.189941 18382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:00.176688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:40465 (local address 127.5.83.2:55805)
0808 04:05:00.177385 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0808 04:05:00.177430 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 04:05:00.179043 (+ 1613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:05:00.179630 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:05:00.179637 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:05:00.180163 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:05:00.180683 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:00.180700 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:00.181703 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:00.181706 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:00.182880 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:00.182892 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:00.183648 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:00.183656 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:00.183957 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:00.184769 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:05:00.184794 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:05:00.188823 (+ 4029us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.62:40465: Server oryx/127.5.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":288,"threads_started":1}
W20260808 04:05:00.190622 18380 heartbeater.cc:644] Failed to heartbeat to 127.5.83.62:40465 (0 consecutive failures): Not authorized: Failed to ping master at 127.5.83.62:40465: Client connection negotiation failed: client connection to 127.5.83.62:40465: Server oryx/127.5.83.62@KRBTEST.COM not found in Kerberos database
Aug 08 04:05:00 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.2@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:05:00 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.2@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:05:00.198331 18382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:00.191151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:40465 (local address 127.5.83.2:33337)
0808 04:05:00.191284 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0808 04:05:00.191301 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:05:00.191393 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:05:00.191668 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:05:00.191672 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:05:00.192006 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:05:00.192134 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:00.192142 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:00.193098 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:00.193102 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:00.193963 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:00.193975 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:00.194623 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:00.194629 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:00.194781 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:00.195326 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:05:00.195349 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:05:00.198049 (+ 2700us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.62:40465: Server oryx/127.5.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":34}
W20260808 04:05:00.198484 18194 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:05:00.191264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.2:33337 (local address 127.5.83.62:40465)
0808 04:05:00.191420 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0808 04:05:00.191426 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:05:00.191442 (+ 16us) server_negotiation.cc:407] Connection header received
0808 04:05:00.191491 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:05:00.191495 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:05:00.191551 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:05:00.191629 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:05:00.192270 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:00.192964 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:00.194098 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:00.194489 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:00.194597 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:00.198269 (+ 3672us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.2:33337: BlockingRecv error: recv got EOF from 127.5.83.2:33337 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 08 04:05:00 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.2@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:05:00 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.2@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:05:00.206342 18382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:00.199072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:40465 (local address 127.5.83.2:51167)
0808 04:05:00.199210 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0808 04:05:00.199227 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:05:00.199315 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:05:00.199595 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:05:00.199599 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:05:00.199911 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:05:00.200048 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:00.200056 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:00.201003 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:00.201008 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:00.201881 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:00.201892 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:00.202530 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:00.202536 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:00.202690 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:00.203221 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:05:00.203244 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:05:00.206079 (+ 2835us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.62:40465: Server oryx/127.5.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":35}
W20260808 04:05:00.206606 18380 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 04:05:00.206771 18194 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:05:00.199183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.2:51167 (local address 127.5.83.62:40465)
0808 04:05:00.199366 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0808 04:05:00.199371 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:05:00.199387 (+ 16us) server_negotiation.cc:407] Connection header received
0808 04:05:00.199429 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:05:00.199432 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:05:00.199481 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:05:00.199555 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:05:00.200180 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:00.200869 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:00.202011 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:00.202401 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:00.202503 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:00.206283 (+ 3780us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.2:51167: BlockingRecv error: recv got EOF from 127.5.83.2:51167 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260808 04:05:00.501269 18383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:05:00.501540 18383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:05:00.501679 18383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:05:00.508574 18383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:05:00.508682 18383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:05:00.508829 18383 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.3
I20260808 04:05:00.517927 18383 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:41297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/krb5kdc/oryx_127.5.83.3.keytab
--principal=oryx/127.5.83.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.3:35613
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.83.3
--webserver_port=37063
--tserver_master_addrs=127.5.83.62:40465
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.3
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:05:00.519088 18383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:05:00.520337 18383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:05:00.532210 18390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:05:00.532174 18397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:05:00.532253 18394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:05:00.533452 18383 server_base.cc:1034] running on GCE node
I20260808 04:05:00.533952 18383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:05:00.534755 18383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:05:00.535930 18383 hybrid_clock.cc:648] HybridClock initialized: now 1786161900535885 us; error 44 us; skew 500 ppm
I20260808 04:05:00.536204 18383 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:05:00 dist-test-slave-1l3l krb5kdc[17088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161900, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.83.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:05:00.539085 18383 init.cc:377] Logged in from keytab as oryx/127.5.83.3@KRBTEST.COM (short username oryx)
I20260808 04:05:00.540514 18383 webserver.cc:460] Webserver started at http://127.5.83.3:37063/ using document root <none> and password file <none>
I20260808 04:05:00.541121 18383 fs_manager.cc:359] Metadata directory not provided
I20260808 04:05:00.541244 18383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:05:00.546233 18383 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 04:05:00.548885 18406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:05:00.550149 18383 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:05:00.550304 18383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-2/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-2/wal
uuid: "4c293b7b37b14b82842c337daf292a69"
format_stamp: "Formatted at 2026-08-08 04:04:58 on dist-test-slave-1l3l"
server_key: "7c99f00aeb7327e504ef0c346eaae098"
server_key_iv: "fa0d3461f6f741ea08e3ee34a531be25"
server_key_version: "encryptionkey@0"
I20260808 04:05:00.551208 18383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:05:00.568660 18383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:05:00.572443 18383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:05:00.572685 18383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:05:00.573634 18383 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:05:00.574878 18383 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:05:00.574975 18383 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:05:00.575083 18383 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:05:00.575150 18383 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:05:00.605805 18383 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.3:35613
I20260808 04:05:00.605865 18578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.3:35613 every 8 connection(s)
I20260808 04:05:00.607206 18383 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
I20260808 04:05:00.609081 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 18383
Aug 08 04:05:00 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.3@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:05:00 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.3@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
W20260808 04:05:00.622675 18194 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:05:00.609873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.3:46963 (local address 127.5.83.62:40465)
0808 04:05:00.610079 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0808 04:05:00.610085 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:05:00.611275 (+ 1190us) server_negotiation.cc:407] Connection header received
0808 04:05:00.613137 (+ 1862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:05:00.613146 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:05:00.613230 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:05:00.613321 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:05:00.614669 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:00.615440 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:00.617115 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:00.617534 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:00.617640 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:00.622406 (+ 4766us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.3:46963: BlockingRecv error: recv got EOF from 127.5.83.3:46963 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
I20260808 04:05:00.623296 18582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:00.610358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:40465 (local address 127.5.83.3:46963)
0808 04:05:00.611113 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0808 04:05:00.611157 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 04:05:00.612910 (+ 1753us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:05:00.613515 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:05:00.613523 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:05:00.613983 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:05:00.614509 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:00.614526 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:00.615598 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:00.615603 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:00.616979 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:00.616991 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:00.617743 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:00.617751 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:00.617967 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:00.618564 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:05:00.618583 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:05:00.622186 (+ 3603us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.62:40465: Server oryx/127.5.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":462,"threads_started":1}
W20260808 04:05:00.623915 18579 heartbeater.cc:644] Failed to heartbeat to 127.5.83.62:40465 (0 consecutive failures): Not authorized: Failed to ping master at 127.5.83.62:40465: Client connection negotiation failed: client connection to 127.5.83.62:40465: Server oryx/127.5.83.62@KRBTEST.COM not found in Kerberos database
I20260808 04:05:00.624835 18584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:00.610738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54736 (local address 127.5.83.62:40465)
0808 04:05:00.611149 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0808 04:05:00.611153 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:05:00.611168 (+ 15us) server_negotiation.cc:407] Connection header received
0808 04:05:00.611221 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:05:00.611225 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:05:00.611291 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:05:00.611387 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:05:00.613024 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:00.613892 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:00.615138 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:00.615559 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:00.615666 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:00.618197 (+ 2531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:05:00.618207 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:05:00.618226 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:05:00.618269 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:05:00.621422 (+ 3153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:05:00.622110 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:05:00.622117 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:05:00.622124 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:05:00.622290 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:05:00.622921 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:05:00.622926 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:05:00.622930 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:05:00.623369 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:05:00.623668 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0808 04:05:00.624120 (+ 452us) server_negotiation.cc:299] Negotiation successful
0808 04:05:00.624448 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":824,"threads_started":1}
Aug 08 04:05:00 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.3@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:05:00 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.3@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:05:00.631747 18582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:00.624446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:40465 (local address 127.5.83.3:50269)
0808 04:05:00.624625 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0808 04:05:00.624707 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0808 04:05:00.624809 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:05:00.625163 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:05:00.625168 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:05:00.625577 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:05:00.625753 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:00.625765 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:00.626837 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:00.626841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:00.627898 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:00.627910 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:00.628558 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:00.628563 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:00.628725 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:00.629207 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:05:00.629223 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:05:00.631483 (+ 2260us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.62:40465: Server oryx/127.5.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":62}
W20260808 04:05:00.631909 18194 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:05:00.624735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.3:50269 (local address 127.5.83.62:40465)
0808 04:05:00.624874 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0808 04:05:00.624877 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:05:00.624901 (+ 24us) server_negotiation.cc:407] Connection header received
0808 04:05:00.624943 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:05:00.624948 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:05:00.625032 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:05:00.625118 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:05:00.625887 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:00.626700 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:00.628023 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:00.628428 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:00.628531 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:00.631664 (+ 3133us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.3:50269: BlockingRecv error: recv got EOF from 127.5.83.3:50269 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Aug 08 04:05:00 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.3@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:05:00 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.3@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:05:00.638517 18582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:00.632316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:40465 (local address 127.5.83.3:47917)
0808 04:05:00.632464 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0808 04:05:00.632483 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 04:05:00.632573 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:05:00.632824 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:05:00.632827 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:05:00.633070 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:05:00.633194 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:00.633201 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:00.634064 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:00.634067 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:00.634913 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:00.634922 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:00.635462 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:00.635466 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:00.635570 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:00.636024 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:05:00.636040 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:05:00.638333 (+ 2293us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.62:40465: Server oryx/127.5.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":43}
W20260808 04:05:00.638757 18579 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 04:05:00.638860 18194 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:05:00.632430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.3:47917 (local address 127.5.83.62:40465)
0808 04:05:00.632634 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0808 04:05:00.632638 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:05:00.632652 (+ 14us) server_negotiation.cc:407] Connection header received
0808 04:05:00.632686 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:05:00.632689 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:05:00.632733 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:05:00.632801 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:05:00.633289 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:00.633951 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:00.635012 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:00.635368 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:00.635450 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:00.638589 (+ 3139us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.3:47917: BlockingRecv error: recv got EOF from 127.5.83.3:47917 (error 108)
Metrics: {"server-negotiator.queue_time_us":120}
Aug 08 04:05:00 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.1@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:05:00 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.1@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:05:00.787401 18585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:00.779776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:40465 (local address 127.5.83.1:47283)
0808 04:05:00.780310 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0808 04:05:00.780326 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:05:00.780440 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:05:00.780693 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:05:00.780696 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:05:00.781159 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:05:00.781363 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:00.781372 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:00.782279 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:00.782283 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:00.783227 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:00.783237 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:00.783979 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:00.783984 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:00.784101 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:00.784631 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:05:00.784648 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:05:00.787181 (+ 2533us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.62:40465: Server oryx/127.5.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":286,"threads_started":1}
W20260808 04:05:00.787599 18194 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:05:00.779823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.1:47283 (local address 127.5.83.62:40465)
0808 04:05:00.780001 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0808 04:05:00.780007 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:05:00.780398 (+ 391us) server_negotiation.cc:407] Connection header received
0808 04:05:00.780531 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:05:00.780535 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:05:00.780596 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:05:00.780663 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:05:00.781485 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:00.782174 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:00.783376 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:00.783828 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:00.783918 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:00.787406 (+ 3488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.1:47283: BlockingRecv error: recv got EOF from 127.5.83.1:47283 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 08 04:05:01 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.2@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:05:01 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.2@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:05:01.215581 18587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:01.207216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:40465 (local address 127.5.83.2:42993)
0808 04:05:01.207753 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0808 04:05:01.207773 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 04:05:01.207925 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:05:01.208245 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:05:01.208250 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:05:01.208738 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:05:01.208949 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:01.208959 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:01.209948 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:01.209952 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:01.210936 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:01.210949 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:01.211742 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:01.211750 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:01.211917 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:01.212490 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:05:01.212508 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:05:01.215258 (+ 2750us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.62:40465: Server oryx/127.5.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":276,"threads_started":1}
W20260808 04:05:01.215773 18194 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:05:01.207305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.2:42993 (local address 127.5.83.62:40465)
0808 04:05:01.207542 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0808 04:05:01.207550 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:05:01.207878 (+ 328us) server_negotiation.cc:407] Connection header received
0808 04:05:01.208045 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:05:01.208050 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:05:01.208112 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:05:01.208193 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:05:01.209100 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:01.209804 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:01.211093 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:01.211527 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:01.211636 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:01.215513 (+ 3877us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.2:42993: BlockingRecv error: recv got EOF from 127.5.83.2:42993 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Aug 08 04:05:01 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.3@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:05:01 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.3@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:05:01.651551 18593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:01.639294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:40465 (local address 127.5.83.3:60399)
0808 04:05:01.639817 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0808 04:05:01.639837 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 04:05:01.639959 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:05:01.640228 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:05:01.640232 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:05:01.640672 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:05:01.640869 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:01.640878 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:01.641946 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:01.641949 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:01.642935 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:01.642947 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:01.643645 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:01.643649 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:01.643851 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:01.648208 (+ 4357us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:05:01.648226 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:05:01.651254 (+ 3028us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.62:40465: Server oryx/127.5.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":259,"threads_started":1}
W20260808 04:05:01.651676 18194 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:05:01.639354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.3:60399 (local address 127.5.83.62:40465)
0808 04:05:01.639540 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0808 04:05:01.639547 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:05:01.639914 (+ 367us) server_negotiation.cc:407] Connection header received
0808 04:05:01.640066 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:05:01.640070 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:05:01.640131 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:05:01.640200 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:05:01.641035 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:01.641819 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:01.643070 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:01.643503 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:01.643593 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:01.651481 (+ 7888us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.3:60399: BlockingRecv error: recv got EOF from 127.5.83.3:60399 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 08 04:05:01 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.1@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:05:01 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.1@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:05:01.796501 18594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:01.788351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:40465 (local address 127.5.83.1:43169)
0808 04:05:01.788892 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0808 04:05:01.788931 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 04:05:01.789062 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:05:01.789410 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:05:01.789415 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:05:01.789926 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:05:01.790083 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:01.790093 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:01.791079 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:01.791083 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:01.792077 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:01.792094 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:01.792750 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:01.792755 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:01.792882 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:01.793450 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:05:01.793469 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:05:01.796253 (+ 2784us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.62:40465: Server oryx/127.5.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":288,"threads_started":1}
W20260808 04:05:01.796643 18194 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:05:01.788447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.1:43169 (local address 127.5.83.62:40465)
0808 04:05:01.788674 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0808 04:05:01.788679 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:05:01.789006 (+ 327us) server_negotiation.cc:407] Connection header received
0808 04:05:01.789171 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:05:01.789177 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:05:01.789256 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:05:01.789353 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:05:01.790206 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:01.790964 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:01.792223 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:01.792624 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:01.792712 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:01.796427 (+ 3715us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.1:43169: BlockingRecv error: recv got EOF from 127.5.83.1:43169 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Aug 08 04:05:02 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.2@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:05:02 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.2@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:05:02.224915 18596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:02.216606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:40465 (local address 127.5.83.2:40505)
0808 04:05:02.217169 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0808 04:05:02.217190 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 04:05:02.217324 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:05:02.217648 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:05:02.217653 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:05:02.218078 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:05:02.218234 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:02.218243 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:02.219268 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:02.219273 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:02.220232 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:02.220245 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:02.220957 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:02.220963 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:02.221113 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:02.221758 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:05:02.221777 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:05:02.224577 (+ 2800us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.62:40465: Server oryx/127.5.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":316,"threads_started":1}
W20260808 04:05:02.225014 18194 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:05:02.216659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.2:40505 (local address 127.5.83.62:40465)
0808 04:05:02.216834 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0808 04:05:02.216840 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:05:02.217277 (+ 437us) server_negotiation.cc:407] Connection header received
0808 04:05:02.217442 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:05:02.217447 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:05:02.217508 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:05:02.217589 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:05:02.218379 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:02.219125 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:02.220375 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:02.220796 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:02.220906 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:02.224792 (+ 3886us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.2:40505: BlockingRecv error: recv got EOF from 127.5.83.2:40505 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 08 04:05:02 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.3@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:05:02 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.3@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:05:02.660948 18598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:02.652522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:40465 (local address 127.5.83.3:35889)
0808 04:05:02.653026 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0808 04:05:02.653042 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:05:02.653166 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:05:02.653431 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:05:02.653434 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:05:02.653800 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:05:02.653969 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:02.653979 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:02.655017 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:02.655021 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:02.656187 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:02.656204 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:02.657075 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:02.657081 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:02.657240 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:02.657837 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:05:02.657855 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:05:02.660621 (+ 2766us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.62:40465: Server oryx/127.5.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":277,"threads_started":1}
W20260808 04:05:02.661072 18194 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:05:02.652582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.3:35889 (local address 127.5.83.62:40465)
0808 04:05:02.652765 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0808 04:05:02.652771 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:05:02.653107 (+ 336us) server_negotiation.cc:407] Connection header received
0808 04:05:02.653260 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:05:02.653263 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:05:02.653324 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:05:02.653401 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:05:02.654113 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:02.654868 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:02.656343 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:02.656932 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:02.657045 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:02.660821 (+ 3776us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.3:35889: BlockingRecv error: recv got EOF from 127.5.83.3:35889 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Aug 08 04:05:02 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.1@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:05:02 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.1@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:05:02.804674 18599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:02.797335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:40465 (local address 127.5.83.1:39741)
0808 04:05:02.797831 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0808 04:05:02.797849 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 04:05:02.797971 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:05:02.798223 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:05:02.798225 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:05:02.798572 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:05:02.798736 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:02.798745 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:02.799678 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:02.799682 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:02.800621 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:02.800633 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:02.801259 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:02.801262 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:02.801379 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:02.801920 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:05:02.801936 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:05:02.804429 (+ 2493us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.62:40465: Server oryx/127.5.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":278,"threads_started":1}
W20260808 04:05:02.804854 18194 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:05:02.797439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.1:39741 (local address 127.5.83.62:40465)
0808 04:05:02.797609 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0808 04:05:02.797614 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:05:02.797923 (+ 309us) server_negotiation.cc:407] Connection header received
0808 04:05:02.798056 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:05:02.798059 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:05:02.798117 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:05:02.798196 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:05:02.798847 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:02.799560 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:02.800728 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:02.801153 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:02.801241 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:02.804621 (+ 3380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.1:39741: BlockingRecv error: recv got EOF from 127.5.83.1:39741 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 08 04:05:03 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.2@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:05:03 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.2@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:05:03.233592 18600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:03.225867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:40465 (local address 127.5.83.2:50835)
0808 04:05:03.226333 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0808 04:05:03.226354 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 04:05:03.226478 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:05:03.226741 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:05:03.226745 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:05:03.227092 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:05:03.227254 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:03.227263 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:03.228344 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:03.228348 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:03.229241 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:03.229253 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:03.229844 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:03.229848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:03.229977 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:03.230560 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:05:03.230576 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:05:03.233330 (+ 2754us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.62:40465: Server oryx/127.5.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":263,"threads_started":1}
W20260808 04:05:03.233757 18194 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:05:03.226025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.2:50835 (local address 127.5.83.62:40465)
0808 04:05:03.226227 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0808 04:05:03.226233 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:05:03.226437 (+ 204us) server_negotiation.cc:407] Connection header received
0808 04:05:03.226572 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:05:03.226576 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:05:03.226638 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:05:03.226707 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:05:03.227383 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:03.228212 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:03.229352 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:03.229738 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:03.229866 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:03.233554 (+ 3688us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.2:50835: BlockingRecv error: recv got EOF from 127.5.83.2:50835 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 08 04:05:03 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.3@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:05:03 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.3@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:05:03.669757 18602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:03.661796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:40465 (local address 127.5.83.3:41177)
0808 04:05:03.662360 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0808 04:05:03.662377 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:05:03.662517 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:05:03.662849 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:05:03.662854 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:05:03.663217 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:05:03.663385 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:03.663395 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:03.664433 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:03.664439 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:03.665374 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:03.665387 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:03.666028 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:03.666034 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:03.666197 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:03.666747 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:05:03.666764 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:05:03.669417 (+ 2653us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.62:40465: Server oryx/127.5.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":283,"threads_started":1}
W20260808 04:05:03.669863 18194 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:05:03.661878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.3:41177 (local address 127.5.83.62:40465)
0808 04:05:03.662062 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0808 04:05:03.662068 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:05:03.662469 (+ 401us) server_negotiation.cc:407] Connection header received
0808 04:05:03.662650 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:05:03.662655 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:05:03.662718 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:05:03.662797 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:05:03.663530 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:03.664306 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:03.665513 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:03.665893 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:03.666002 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:03.669658 (+ 3656us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.3:41177: BlockingRecv error: recv got EOF from 127.5.83.3:41177 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 08 04:05:03 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.1@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:05:03 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.1@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:05:03.813426 18603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:03.805506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:40465 (local address 127.5.83.1:39929)
0808 04:05:03.805955 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0808 04:05:03.805972 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:05:03.806082 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:05:03.806332 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:05:03.806335 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:05:03.806710 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:05:03.806860 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:03.806868 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:03.807803 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:03.807806 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:03.808758 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:03.808768 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:03.809414 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:03.809419 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:03.809562 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:03.810138 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:05:03.810156 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:05:03.813196 (+ 3040us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.62:40465: Server oryx/127.5.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":246,"threads_started":1}
W20260808 04:05:03.813624 18194 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:05:03.805583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.1:39929 (local address 127.5.83.62:40465)
0808 04:05:03.805813 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0808 04:05:03.805820 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:05:03.806045 (+ 225us) server_negotiation.cc:407] Connection header received
0808 04:05:03.806167 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:05:03.806171 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:05:03.806231 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:05:03.806307 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:05:03.806965 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:03.807694 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:03.808859 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:03.809268 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:03.809353 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:03.813376 (+ 4023us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.1:39929: BlockingRecv error: recv got EOF from 127.5.83.1:39929 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Aug 08 04:05:04 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.2@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:05:04 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.2@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:05:04.242769 18606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:04.234466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:40465 (local address 127.5.83.2:46355)
0808 04:05:04.234946 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0808 04:05:04.234969 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 04:05:04.235096 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:05:04.235456 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:05:04.235461 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:05:04.235947 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:05:04.236123 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:04.236133 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:04.237158 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:04.237163 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:04.238129 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:04.238141 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:04.238937 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:04.238943 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:04.239134 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:04.239750 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:05:04.239771 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:05:04.242460 (+ 2689us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.62:40465: Server oryx/127.5.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":257,"threads_started":1}
W20260808 04:05:04.242892 18194 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:05:04.234572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.2:46355 (local address 127.5.83.62:40465)
0808 04:05:04.234780 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0808 04:05:04.234786 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:05:04.235045 (+ 259us) server_negotiation.cc:407] Connection header received
0808 04:05:04.235201 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:05:04.235204 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:05:04.235263 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:05:04.235351 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:05:04.236269 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:04.237029 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:04.238278 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:04.238792 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:04.238907 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:04.242671 (+ 3764us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.2:46355: BlockingRecv error: recv got EOF from 127.5.83.2:46355 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Aug 08 04:05:04 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.3@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:05:04 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.3@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:05:04.678893 18608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:04.670655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:40465 (local address 127.5.83.3:56809)
0808 04:05:04.671182 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0808 04:05:04.671199 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:05:04.671334 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:05:04.671677 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:05:04.671682 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:05:04.672110 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:05:04.672259 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:04.672268 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:04.673289 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:04.673294 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:04.674262 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:04.674274 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:04.675111 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:04.675117 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:04.675314 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:04.675879 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:05:04.675896 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:05:04.678672 (+ 2776us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.62:40465: Server oryx/127.5.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":302,"threads_started":1}
W20260808 04:05:04.679080 18194 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:05:04.670770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.3:56809 (local address 127.5.83.62:40465)
0808 04:05:04.670968 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0808 04:05:04.670976 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:05:04.671284 (+ 308us) server_negotiation.cc:407] Connection header received
0808 04:05:04.671471 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:05:04.671477 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:05:04.671547 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:05:04.671635 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:05:04.672404 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:04.673162 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:04.674402 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:04.674972 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:04.675088 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:04.678858 (+ 3770us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.3:56809: BlockingRecv error: recv got EOF from 127.5.83.3:56809 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 08 04:05:04 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.1@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:05:04 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.1@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:05:04.822275 18609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:04.814281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:40465 (local address 127.5.83.1:45811)
0808 04:05:04.814829 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0808 04:05:04.814846 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:05:04.814974 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:05:04.815228 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:05:04.815231 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:05:04.815692 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:05:04.815926 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:04.815936 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:04.816861 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:04.816865 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:04.817880 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:04.817893 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:04.818617 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:04.818622 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:04.818781 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:04.819386 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:05:04.819405 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:05:04.822033 (+ 2628us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.62:40465: Server oryx/127.5.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":278,"threads_started":1}
W20260808 04:05:04.822396 18194 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:05:04.814355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.1:45811 (local address 127.5.83.62:40465)
0808 04:05:04.814534 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0808 04:05:04.814538 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:05:04.814913 (+ 375us) server_negotiation.cc:407] Connection header received
0808 04:05:04.815061 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:05:04.815065 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:05:04.815120 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:05:04.815201 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:05:04.816063 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:04.816756 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:04.818009 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:04.818474 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:04.818598 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:04.822209 (+ 3611us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.1:45811: BlockingRecv error: recv got EOF from 127.5.83.1:45811 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 08 04:05:05 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.2@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:05:05 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.2@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:05:05.252182 18610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:05.243812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:40465 (local address 127.5.83.2:47893)
0808 04:05:05.244309 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0808 04:05:05.244330 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 04:05:05.244471 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:05:05.244782 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:05:05.244786 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:05:05.245206 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:05:05.245377 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:05.245415 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:05.246340 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:05.246343 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:05.247365 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:05.247391 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:05.248353 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:05.248359 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:05.248527 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:05.249092 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:05:05.249109 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:05:05.251927 (+ 2818us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.62:40465: Server oryx/127.5.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":252,"threads_started":1}
W20260808 04:05:05.252292 18194 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:05:05.243949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.2:47893 (local address 127.5.83.62:40465)
0808 04:05:05.244158 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0808 04:05:05.244164 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:05:05.244405 (+ 241us) server_negotiation.cc:407] Connection header received
0808 04:05:05.244596 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:05:05.244599 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:05:05.244662 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:05:05.244732 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:05:05.245518 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:05.246233 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:05.247520 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:05.248192 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:05.248313 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:05.252085 (+ 3772us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.2:47893: BlockingRecv error: recv got EOF from 127.5.83.2:47893 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Aug 08 04:05:05 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.3@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:05:05 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.3@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:05:05.688135 18612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:05.679797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:40465 (local address 127.5.83.3:38537)
0808 04:05:05.680261 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0808 04:05:05.680277 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:05:05.680389 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:05:05.680683 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:05:05.680686 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:05:05.681088 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:05:05.681252 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:05.681261 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:05.682238 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:05.682243 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:05.683201 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:05.683214 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:05.683916 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:05.683923 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:05.684099 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:05.684846 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:05:05.684867 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:05:05.687831 (+ 2964us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.62:40465: Server oryx/127.5.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":222,"threads_started":1}
W20260808 04:05:05.688232 18194 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:05:05.679828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.3:38537 (local address 127.5.83.62:40465)
0808 04:05:05.680020 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0808 04:05:05.680027 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:05:05.680364 (+ 337us) server_negotiation.cc:407] Connection header received
0808 04:05:05.680521 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:05:05.680525 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:05:05.680585 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:05:05.680653 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:05:05.681355 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:05.682081 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:05.683315 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:05.683766 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:05.683872 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:05.688020 (+ 4148us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.3:38537: BlockingRecv error: recv got EOF from 127.5.83.3:38537 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 08 04:05:05 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.1@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:05:05 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.1@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:05:05.831154 18613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:05.823159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:40465 (local address 127.5.83.1:39479)
0808 04:05:05.823755 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0808 04:05:05.823772 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:05:05.823884 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:05:05.824130 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:05:05.824133 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:05:05.824535 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:05:05.824768 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:05.824781 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:05.826061 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:05.826067 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:05.826985 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:05.826997 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:05.827647 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:05.827651 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:05.827826 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:05.828364 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:05:05.828387 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:05:05.830936 (+ 2549us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.62:40465: Server oryx/127.5.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":340,"threads_started":1}
W20260808 04:05:05.831439 18194 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:05:05.823236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.1:39479 (local address 127.5.83.62:40465)
0808 04:05:05.823422 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0808 04:05:05.823427 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:05:05.823833 (+ 406us) server_negotiation.cc:407] Connection header received
0808 04:05:05.823965 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:05:05.823968 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:05:05.824027 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:05:05.824103 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:05:05.824930 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:05.825915 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:05.827107 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:05.827527 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:05.827627 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:05.831145 (+ 3518us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.1:39479: BlockingRecv error: recv got EOF from 127.5.83.1:39479 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 08 04:05:06 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.2@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:05:06 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.2@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:05:06.261371 18614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:06.253109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:40465 (local address 127.5.83.2:45903)
0808 04:05:06.253626 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0808 04:05:06.253646 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 04:05:06.253773 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:05:06.254100 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:05:06.254105 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:05:06.254599 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:05:06.254811 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:06.254825 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:06.256028 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:06.256034 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:06.256975 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:06.256988 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:06.257689 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:06.257695 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:06.257858 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:06.258536 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:05:06.258554 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:05:06.261157 (+ 2603us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.62:40465: Server oryx/127.5.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":306,"threads_started":1}
W20260808 04:05:06.261584 18194 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:05:06.253169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.2:45903 (local address 127.5.83.62:40465)
0808 04:05:06.253337 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0808 04:05:06.253343 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:05:06.253728 (+ 385us) server_negotiation.cc:407] Connection header received
0808 04:05:06.253880 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:05:06.253885 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:05:06.253962 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:05:06.254075 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:05:06.254971 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:06.255891 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:06.257107 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:06.257543 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:06.257654 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:06.261363 (+ 3709us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.2:45903: BlockingRecv error: recv got EOF from 127.5.83.2:45903 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Aug 08 04:05:06 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.3@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:05:06 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.3@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:05:06.701411 18616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:06.689001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:40465 (local address 127.5.83.3:58927)
0808 04:05:06.689520 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0808 04:05:06.689537 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:05:06.689650 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:05:06.689956 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:05:06.689961 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:05:06.690337 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:05:06.690516 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:06.690526 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:06.691539 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:06.691542 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:06.692447 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:06.692457 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:06.693095 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:06.693099 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:06.693209 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:06.698142 (+ 4933us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:05:06.698168 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:05:06.701095 (+ 2927us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.62:40465: Server oryx/127.5.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":275,"threads_started":1}
W20260808 04:05:06.701543 18194 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:05:06.689056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.3:58927 (local address 127.5.83.62:40465)
0808 04:05:06.689209 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0808 04:05:06.689213 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:05:06.689635 (+ 422us) server_negotiation.cc:407] Connection header received
0808 04:05:06.689748 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:05:06.689751 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:05:06.689815 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:05:06.689906 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:05:06.690647 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:06.691428 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:06.692551 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:06.692982 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:06.693070 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:06.701313 (+ 8243us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.3:58927: BlockingRecv error: recv got EOF from 127.5.83.3:58927 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 08 04:05:06 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.1@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:05:06 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.1@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:05:06.840452 18617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:06.832107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:40465 (local address 127.5.83.1:54161)
0808 04:05:06.832647 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0808 04:05:06.832664 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:05:06.832778 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:05:06.833051 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:05:06.833054 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:05:06.833476 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:05:06.833655 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:06.833664 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:06.834578 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:06.834582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:06.835511 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:06.835522 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:06.836222 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:06.836228 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:06.836388 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:06.836986 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:05:06.837003 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:05:06.840199 (+ 3196us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.62:40465: Server oryx/127.5.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":232,"threads_started":1}
W20260808 04:05:06.840591 18194 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:05:06.832165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.1:54161 (local address 127.5.83.62:40465)
0808 04:05:06.832358 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0808 04:05:06.832366 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:05:06.832738 (+ 372us) server_negotiation.cc:407] Connection header received
0808 04:05:06.832863 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:05:06.832866 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:05:06.832947 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:05:06.833024 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:05:06.833761 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:06.834470 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:06.835619 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:06.836064 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:06.836161 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:06.840408 (+ 4247us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.1:54161: BlockingRecv error: recv got EOF from 127.5.83.1:54161 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 08 04:05:07 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.2@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:05:07 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.2@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:05:07.271179 18618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:07.262470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:40465 (local address 127.5.83.2:48453)
0808 04:05:07.263132 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0808 04:05:07.263153 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 04:05:07.263290 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:05:07.263617 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:05:07.263621 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:05:07.264097 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:05:07.264279 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:07.264289 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:07.265355 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:07.265359 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:07.266276 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:07.266288 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:07.267090 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:07.267096 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:07.267262 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:07.267880 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:05:07.267904 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:05:07.270839 (+ 2935us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.62:40465: Server oryx/127.5.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":236,"threads_started":1}
W20260808 04:05:07.271268 18194 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:05:07.262452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.2:48453 (local address 127.5.83.62:40465)
0808 04:05:07.262680 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0808 04:05:07.262684 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:05:07.263246 (+ 562us) server_negotiation.cc:407] Connection header received
0808 04:05:07.263414 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:05:07.263419 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:05:07.263482 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:05:07.263562 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:05:07.264439 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:07.265210 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:07.266429 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:07.266943 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:07.267050 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:07.271042 (+ 3992us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.2:48453: BlockingRecv error: recv got EOF from 127.5.83.2:48453 (error 108)
Metrics: {"server-negotiator.queue_time_us":109}
Aug 08 04:05:07 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.3@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:05:07 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.3@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:05:07.710093 18619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:07.702350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:40465 (local address 127.5.83.3:60465)
0808 04:05:07.702957 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0808 04:05:07.702974 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:05:07.703089 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:05:07.703351 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:05:07.703354 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:05:07.703771 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:05:07.703957 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:07.703966 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:07.704908 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:07.704912 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:07.705808 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:07.705818 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:07.706467 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:07.706471 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:07.706578 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:07.707164 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:05:07.707181 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:05:07.709839 (+ 2658us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.62:40465: Server oryx/127.5.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":255,"threads_started":1}
W20260808 04:05:07.710192 18194 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:05:07.702435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.3:60465 (local address 127.5.83.62:40465)
0808 04:05:07.702595 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0808 04:05:07.702600 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:05:07.703060 (+ 460us) server_negotiation.cc:407] Connection header received
0808 04:05:07.703183 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:05:07.703187 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:05:07.703247 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:05:07.703323 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:05:07.704073 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:07.704792 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:07.705927 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:07.706347 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:07.706444 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:07.710025 (+ 3581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.3:60465: BlockingRecv error: recv got EOF from 127.5.83.3:60465 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 08 04:05:07 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.1@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:05:07 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.1@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:05:07.849252 18621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:07.841315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:40465 (local address 127.5.83.1:33233)
0808 04:05:07.841825 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0808 04:05:07.841841 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:05:07.841966 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:05:07.842268 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:05:07.842271 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:05:07.842676 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:05:07.842831 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:07.842840 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:07.843825 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:07.843829 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:07.844788 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:07.844801 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:07.845515 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:07.845519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:07.845634 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:07.846204 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:05:07.846222 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:05:07.848988 (+ 2766us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.62:40465: Server oryx/127.5.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":279,"threads_started":1}
W20260808 04:05:07.849419 18194 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:05:07.841471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.1:33233 (local address 127.5.83.62:40465)
0808 04:05:07.841631 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0808 04:05:07.841635 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:05:07.841915 (+ 280us) server_negotiation.cc:407] Connection header received
0808 04:05:07.842076 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:05:07.842080 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:05:07.842157 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:05:07.842235 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:05:07.842954 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:07.843688 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:07.844931 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:07.845374 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:07.845462 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:07.849183 (+ 3721us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.1:33233: BlockingRecv error: recv got EOF from 127.5.83.1:33233 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 08 04:05:08 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.2@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:05:08 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.2@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:05:08.280556 18623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:08.272188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:40465 (local address 127.5.83.2:59815)
0808 04:05:08.272677 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0808 04:05:08.272699 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 04:05:08.272814 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:05:08.273082 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:05:08.273085 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:05:08.273523 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:05:08.273684 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:08.273693 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:08.274837 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:08.274843 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:08.275923 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:08.275942 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:08.276786 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:08.276790 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:08.276916 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:08.277506 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:05:08.277524 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:05:08.280331 (+ 2807us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.62:40465: Server oryx/127.5.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":264,"threads_started":1}
W20260808 04:05:08.280709 18194 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:05:08.272320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.2:59815 (local address 127.5.83.62:40465)
0808 04:05:08.272485 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0808 04:05:08.272489 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:05:08.272763 (+ 274us) server_negotiation.cc:407] Connection header received
0808 04:05:08.272913 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:05:08.272917 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:05:08.272977 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:05:08.273047 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:05:08.273793 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:08.274670 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:08.276117 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:08.276624 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:08.276720 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:08.280515 (+ 3795us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.2:59815: BlockingRecv error: recv got EOF from 127.5.83.2:59815 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 08 04:05:08 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.3@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:05:08 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.3@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:05:08.719308 18624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:08.711003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:40465 (local address 127.5.83.3:55253)
0808 04:05:08.711516 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0808 04:05:08.711535 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 04:05:08.711656 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:05:08.711976 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:05:08.711980 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:05:08.712392 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:05:08.712555 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:08.712563 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:08.713655 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:08.713661 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:08.714744 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:08.714764 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:08.715771 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:08.715780 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:08.715919 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:08.716475 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:05:08.716493 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:05:08.719076 (+ 2583us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.62:40465: Server oryx/127.5.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":281,"threads_started":1}
W20260808 04:05:08.719447 18194 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:05:08.711083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.3:55253 (local address 127.5.83.62:40465)
0808 04:05:08.711260 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0808 04:05:08.711265 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:05:08.711603 (+ 338us) server_negotiation.cc:407] Connection header received
0808 04:05:08.711790 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:05:08.711793 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:05:08.711854 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:05:08.711946 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:05:08.712710 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:08.713483 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:08.714921 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:08.715544 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:08.715668 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:08.719256 (+ 3588us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.3:55253: BlockingRecv error: recv got EOF from 127.5.83.3:55253 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 08 04:05:08 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.1@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:05:08 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.1@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:05:08.858727 18625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:08.850070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:40465 (local address 127.5.83.1:47515)
0808 04:05:08.850601 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0808 04:05:08.850616 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:05:08.850731 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:05:08.851019 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:05:08.851022 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:05:08.851426 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:05:08.851606 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:08.851615 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:08.852781 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:08.852784 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:08.853963 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:08.853980 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:08.854739 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:08.854745 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:08.854916 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:08.855514 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:05:08.855532 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:05:08.858474 (+ 2942us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.62:40465: Server oryx/127.5.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":279,"threads_started":1}
W20260808 04:05:08.858865 18194 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:05:08.850120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.1:47515 (local address 127.5.83.62:40465)
0808 04:05:08.850267 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0808 04:05:08.850271 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:05:08.850713 (+ 442us) server_negotiation.cc:407] Connection header received
0808 04:05:08.850824 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:05:08.850829 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:05:08.850910 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:05:08.850991 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:05:08.851778 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:08.852647 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:08.854083 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:08.854590 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:08.854700 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:08.858655 (+ 3955us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.1:47515: BlockingRecv error: recv got EOF from 127.5.83.1:47515 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Aug 08 04:05:09 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.2@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:05:09 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.2@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:05:09.290946 18628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:09.281506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:40465 (local address 127.5.83.2:33611)
0808 04:05:09.282006 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0808 04:05:09.282026 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 04:05:09.282141 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:05:09.282390 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:05:09.282393 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:05:09.282784 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:05:09.282963 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:09.282971 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:09.284448 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:09.284453 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:09.285993 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:09.286009 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:09.286687 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:09.286693 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:09.286873 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:09.287463 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:05:09.287486 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:05:09.290611 (+ 3125us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.62:40465: Server oryx/127.5.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":262,"threads_started":1}
W20260808 04:05:09.291045 18194 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:05:09.281581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.2:33611 (local address 127.5.83.62:40465)
0808 04:05:09.281778 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0808 04:05:09.281786 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:05:09.282098 (+ 312us) server_negotiation.cc:407] Connection header received
0808 04:05:09.282222 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:05:09.282226 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:05:09.282287 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:05:09.282363 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:05:09.283113 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:09.284268 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:09.286126 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:09.286550 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:09.286641 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:09.290812 (+ 4171us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.2:33611: BlockingRecv error: recv got EOF from 127.5.83.2:33611 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 08 04:05:09 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.3@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:05:09 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.3@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:05:09.728451 18630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:09.720240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:40465 (local address 127.5.83.3:44695)
0808 04:05:09.720769 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0808 04:05:09.720785 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:05:09.720899 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:05:09.721179 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:05:09.721182 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:05:09.721585 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:05:09.721760 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:09.721769 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:09.722950 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:09.722955 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:09.724012 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:09.724026 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:09.724703 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:09.724706 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:09.724823 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:09.725424 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:05:09.725442 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:05:09.728202 (+ 2760us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.62:40465: Server oryx/127.5.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":290,"threads_started":1}
W20260808 04:05:09.728572 18194 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:05:09.720318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.3:44695 (local address 127.5.83.62:40465)
0808 04:05:09.720495 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0808 04:05:09.720499 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:05:09.720852 (+ 353us) server_negotiation.cc:407] Connection header received
0808 04:05:09.721010 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:05:09.721014 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:05:09.721077 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:05:09.721146 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:05:09.721887 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:09.722821 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:09.724137 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:09.724587 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:09.724684 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:09.728392 (+ 3708us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.3:44695: BlockingRecv error: recv got EOF from 127.5.83.3:44695 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 08 04:05:09 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.1@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:05:09 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.1@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:05:09.867739 18631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:09.859560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:40465 (local address 127.5.83.1:44357)
0808 04:05:09.860093 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0808 04:05:09.860110 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:05:09.860224 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:05:09.860567 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:05:09.860570 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:05:09.860945 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:05:09.861147 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:09.861159 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:09.862440 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:09.862444 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:09.863356 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:09.863369 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:09.864113 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:09.864119 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:09.864251 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:09.864845 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:05:09.864864 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:05:09.867490 (+ 2626us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.62:40465: Server oryx/127.5.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":313,"threads_started":1}
W20260808 04:05:09.867925 18194 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:05:09.859643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.1:44357 (local address 127.5.83.62:40465)
0808 04:05:09.859831 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0808 04:05:09.859835 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:05:09.860177 (+ 342us) server_negotiation.cc:407] Connection header received
0808 04:05:09.860370 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:05:09.860374 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:05:09.860457 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:05:09.860545 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:05:09.861299 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:09.862268 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:09.863493 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:09.863959 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:09.864047 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:09.867691 (+ 3644us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.1:44357: BlockingRecv error: recv got EOF from 127.5.83.1:44357 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Aug 08 04:05:10 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.2@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:05:10 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.2@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:05:10.299558 18633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:10.291897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:40465 (local address 127.5.83.2:45533)
0808 04:05:10.292371 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0808 04:05:10.292423 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 04:05:10.292534 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:05:10.292781 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:05:10.292784 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:05:10.293142 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:05:10.293284 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:10.293293 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:10.294220 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:10.294224 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:10.295195 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:10.295207 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:10.295931 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:10.295936 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:10.296056 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:10.296668 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:05:10.296686 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:05:10.299318 (+ 2632us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.62:40465: Server oryx/127.5.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":272,"threads_started":1}
W20260808 04:05:10.299780 18194 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:05:10.291905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.2:45533 (local address 127.5.83.62:40465)
0808 04:05:10.292057 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0808 04:05:10.292061 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:05:10.292484 (+ 423us) server_negotiation.cc:407] Connection header received
0808 04:05:10.292614 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:05:10.292617 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:05:10.292674 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:05:10.292747 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:05:10.293393 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:10.294101 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:10.295325 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:10.295806 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:10.295895 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:10.299526 (+ 3631us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.2:45533: BlockingRecv error: recv got EOF from 127.5.83.2:45533 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Aug 08 04:05:10 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.3@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:05:10 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.3@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:05:10.737344 18634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:10.729444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:40465 (local address 127.5.83.3:33507)
0808 04:05:10.729986 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0808 04:05:10.730006 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 04:05:10.730135 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:05:10.730489 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:05:10.730494 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:05:10.730870 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:05:10.731016 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:10.731025 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:10.732061 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:10.732067 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:10.733004 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:10.733015 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:10.733679 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:10.733684 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:10.733848 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:10.734368 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:05:10.734384 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:05:10.737047 (+ 2663us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.62:40465: Server oryx/127.5.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":304,"threads_started":1}
W20260808 04:05:10.737504 18194 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:05:10.729602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.3:33507 (local address 127.5.83.62:40465)
0808 04:05:10.729838 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0808 04:05:10.729844 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:05:10.730086 (+ 242us) server_negotiation.cc:407] Connection header received
0808 04:05:10.730257 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:05:10.730261 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:05:10.730321 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:05:10.730407 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:05:10.731156 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:10.731935 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:10.733144 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:10.733537 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:10.733642 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:10.737273 (+ 3631us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.3:33507: BlockingRecv error: recv got EOF from 127.5.83.3:33507 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 08 04:05:10 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.1@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:05:10 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.1@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:05:10.876859 18635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:10.868667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:40465 (local address 127.5.83.1:57835)
0808 04:05:10.869201 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0808 04:05:10.869217 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:05:10.869351 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:05:10.869706 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:05:10.869712 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:05:10.870158 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:05:10.870336 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:10.870345 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:10.871394 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:10.871398 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:10.872309 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:10.872320 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:10.873027 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:10.873034 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:10.873188 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:10.873788 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:05:10.873806 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:05:10.876534 (+ 2728us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.62:40465: Server oryx/127.5.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":304,"threads_started":1}
W20260808 04:05:10.876969 18194 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:05:10.868760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.1:57835 (local address 127.5.83.62:40465)
0808 04:05:10.868975 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0808 04:05:10.868981 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:05:10.869303 (+ 322us) server_negotiation.cc:407] Connection header received
0808 04:05:10.869480 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:05:10.869485 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:05:10.869569 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:05:10.869651 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:05:10.870486 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:10.871248 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:10.872456 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:10.872880 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:10.872987 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:10.876749 (+ 3762us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.1:57835: BlockingRecv error: recv got EOF from 127.5.83.1:57835 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Aug 08 04:05:11 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.2@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:05:11 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.2@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:05:11.308590 18639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:11.300506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:40465 (local address 127.5.83.2:49949)
0808 04:05:11.301009 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0808 04:05:11.301029 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 04:05:11.301141 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:05:11.301404 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:05:11.301408 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:05:11.301807 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:05:11.302005 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:11.302014 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:11.302972 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:11.302976 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:11.303968 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:11.303981 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:11.304821 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:11.304827 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:11.304996 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:11.305579 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:05:11.305595 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:05:11.308345 (+ 2750us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.62:40465: Server oryx/127.5.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":265,"threads_started":1}
W20260808 04:05:11.308848 18194 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:05:11.300579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.2:49949 (local address 127.5.83.62:40465)
0808 04:05:11.300744 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0808 04:05:11.300748 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:05:11.301090 (+ 342us) server_negotiation.cc:407] Connection header received
0808 04:05:11.301239 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:05:11.301243 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:05:11.301301 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:05:11.301369 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:05:11.302109 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:11.302849 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:11.304077 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:11.304664 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:11.304782 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:11.308542 (+ 3760us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.2:49949: BlockingRecv error: recv got EOF from 127.5.83.2:49949 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 08 04:05:11 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.3@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:05:11 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.3@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:05:11.748943 18640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:11.738192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:40465 (local address 127.5.83.3:46177)
0808 04:05:11.738691 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0808 04:05:11.738708 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:05:11.738820 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:05:11.739082 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:05:11.739085 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:05:11.739499 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:05:11.739673 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:11.739681 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:11.740655 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:11.740659 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:11.741546 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:11.741556 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:11.742261 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:11.742267 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:11.742442 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:11.745994 (+ 3552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:05:11.746013 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:05:11.748708 (+ 2695us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.62:40465: Server oryx/127.5.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":280,"threads_started":1}
W20260808 04:05:11.749082 18194 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:05:11.738274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.3:46177 (local address 127.5.83.62:40465)
0808 04:05:11.738501 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0808 04:05:11.738507 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:05:11.738780 (+ 273us) server_negotiation.cc:407] Connection header received
0808 04:05:11.738913 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:05:11.738917 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:05:11.738977 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:05:11.739054 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:05:11.739825 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:11.740550 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:11.741644 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:11.742104 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:11.742219 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:11.748878 (+ 6659us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.3:46177: BlockingRecv error: recv got EOF from 127.5.83.3:46177 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 08 04:05:11 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.1@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:05:11 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.1@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:05:11.886099 18641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:11.877797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:40465 (local address 127.5.83.1:39321)
0808 04:05:11.878362 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0808 04:05:11.878379 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:05:11.878501 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:05:11.878781 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:05:11.878784 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:05:11.879200 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:05:11.879358 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:11.879367 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:11.880353 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:11.880357 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:11.881272 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:11.881286 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:11.882242 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:11.882247 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:11.882391 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:11.882934 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:05:11.882951 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:05:11.885772 (+ 2821us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.62:40465: Server oryx/127.5.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":315,"threads_started":1}
W20260808 04:05:11.886200 18194 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:05:11.877850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.1:39321 (local address 127.5.83.62:40465)
0808 04:05:11.878014 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0808 04:05:11.878018 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:05:11.878453 (+ 435us) server_negotiation.cc:407] Connection header received
0808 04:05:11.878581 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:05:11.878584 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:05:11.878660 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:05:11.878739 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:05:11.879481 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:11.880224 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:11.881392 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:11.882071 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:11.882190 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:11.885983 (+ 3793us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.1:39321: BlockingRecv error: recv got EOF from 127.5.83.1:39321 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 08 04:05:12 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.2@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:05:12 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.2@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:05:12.317940 18642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:12.309591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:40465 (local address 127.5.83.2:48373)
0808 04:05:12.310266 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0808 04:05:12.310302 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0808 04:05:12.310448 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:05:12.310767 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:05:12.310770 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:05:12.311171 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:05:12.311335 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:12.311344 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:12.312308 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:12.312313 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:12.313234 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:12.313248 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:12.314077 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:12.314082 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:12.314228 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:12.314833 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:05:12.314850 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:05:12.317702 (+ 2852us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.62:40465: Server oryx/127.5.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":415,"threads_started":1}
W20260808 04:05:12.318095 18194 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:05:12.309619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.2:48373 (local address 127.5.83.62:40465)
0808 04:05:12.309844 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0808 04:05:12.309851 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:05:12.310393 (+ 542us) server_negotiation.cc:407] Connection header received
0808 04:05:12.310574 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:05:12.310579 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:05:12.310651 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:05:12.310728 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:05:12.311452 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:12.312168 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:12.313416 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:12.313858 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:12.313962 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:12.317899 (+ 3937us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.2:48373: BlockingRecv error: recv got EOF from 127.5.83.2:48373 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 08 04:05:12 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.3@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:05:12 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.3@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:05:12.757880 18645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:12.749938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:40465 (local address 127.5.83.3:43321)
0808 04:05:12.750405 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0808 04:05:12.750420 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:05:12.750550 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:05:12.750872 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:05:12.750877 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:05:12.751270 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:05:12.751430 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:12.751438 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:12.752544 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:12.752550 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:12.753497 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:12.753509 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:12.754188 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:12.754193 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:12.754359 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:12.754921 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:05:12.754939 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:05:12.757649 (+ 2710us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.62:40465: Server oryx/127.5.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":238,"threads_started":1}
W20260808 04:05:12.758086 18194 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:05:12.749965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.3:43321 (local address 127.5.83.62:40465)
0808 04:05:12.750136 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0808 04:05:12.750142 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:05:12.750500 (+ 358us) server_negotiation.cc:407] Connection header received
0808 04:05:12.750670 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:05:12.750675 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:05:12.750741 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:05:12.750818 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:05:12.751566 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:12.752381 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:12.753627 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:12.754055 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:12.754162 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:12.757848 (+ 3686us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.3:43321: BlockingRecv error: recv got EOF from 127.5.83.3:43321 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Aug 08 04:05:12 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.1@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:05:12 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.1@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:05:12.895618 18646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:12.887194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:40465 (local address 127.5.83.1:51099)
0808 04:05:12.887787 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0808 04:05:12.887805 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 04:05:12.887942 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:05:12.888429 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:05:12.888432 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:05:12.889019 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:05:12.889198 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:12.889208 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:12.890196 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:12.890200 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:12.891166 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:12.891180 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:12.891895 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:12.891901 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:12.892018 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:12.892637 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:05:12.892655 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:05:12.895355 (+ 2700us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.62:40465: Server oryx/127.5.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":298,"threads_started":1}
W20260808 04:05:12.895857 18194 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:05:12.887273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.1:51099 (local address 127.5.83.62:40465)
0808 04:05:12.887513 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0808 04:05:12.887519 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:05:12.887884 (+ 365us) server_negotiation.cc:407] Connection header received
0808 04:05:12.888093 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:05:12.888098 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:05:12.888193 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:05:12.888286 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:05:12.889330 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:12.890065 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:12.891279 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:12.891725 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:12.891847 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:12.895594 (+ 3747us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.1:51099: BlockingRecv error: recv got EOF from 127.5.83.1:51099 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Aug 08 04:05:13 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.2@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:05:13 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.2@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:05:13.327286 18647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:13.318853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:40465 (local address 127.5.83.2:56401)
0808 04:05:13.319388 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0808 04:05:13.319407 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 04:05:13.319539 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:05:13.319868 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:05:13.319872 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:05:13.320305 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:05:13.320474 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:13.320484 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:13.321484 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:13.321489 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:13.322571 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:13.322584 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:13.323305 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:13.323311 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:13.323466 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:13.324036 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:05:13.324054 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:05:13.326970 (+ 2916us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.62:40465: Server oryx/127.5.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":266,"threads_started":1}
W20260808 04:05:13.327404 18194 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:05:13.319021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.2:56401 (local address 127.5.83.62:40465)
0808 04:05:13.319220 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0808 04:05:13.319226 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:05:13.319490 (+ 264us) server_negotiation.cc:407] Connection header received
0808 04:05:13.319647 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:05:13.319651 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:05:13.319731 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:05:13.319811 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:05:13.320611 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:13.321347 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:13.322707 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:13.323134 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:13.323248 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:13.327183 (+ 3935us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.2:56401: BlockingRecv error: recv got EOF from 127.5.83.2:56401 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 08 04:05:13 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.3@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:05:13 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.3@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:05:13.766393 18648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:13.758720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:40465 (local address 127.5.83.3:38505)
0808 04:05:13.759213 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0808 04:05:13.759229 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:05:13.759384 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:05:13.759670 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:05:13.759673 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:05:13.760115 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:05:13.760273 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:13.760283 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:13.761293 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:13.761296 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:13.762268 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:13.762280 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:13.762932 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:13.762936 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:13.763061 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:13.763605 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:05:13.763624 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:05:13.766171 (+ 2547us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.62:40465: Server oryx/127.5.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":250,"threads_started":1}
W20260808 04:05:13.766542 18194 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:05:13.758797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.3:38505 (local address 127.5.83.62:40465)
0808 04:05:13.759008 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0808 04:05:13.759014 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:05:13.759312 (+ 298us) server_negotiation.cc:407] Connection header received
0808 04:05:13.759480 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:05:13.759484 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:05:13.759545 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:05:13.759620 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:05:13.760448 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:13.761165 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:13.762386 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:13.762784 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:13.762870 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:13.766360 (+ 3490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.3:38505: BlockingRecv error: recv got EOF from 127.5.83.3:38505 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 08 04:05:13 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.1@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:05:13 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.1@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:05:13.904282 18650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:13.896511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:40465 (local address 127.5.83.1:44839)
0808 04:05:13.897003 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0808 04:05:13.897021 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 04:05:13.897135 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:05:13.897433 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:05:13.897436 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:05:13.897804 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:05:13.898008 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:13.898017 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:13.899035 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:13.899039 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:13.900036 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:13.900050 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:13.900674 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:13.900679 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:13.900798 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:13.901381 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:05:13.901398 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:05:13.904035 (+ 2637us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.62:40465: Server oryx/127.5.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":246,"threads_started":1}
W20260808 04:05:13.904444 18194 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:05:13.896548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.1:44839 (local address 127.5.83.62:40465)
0808 04:05:13.896794 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0808 04:05:13.896798 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:05:13.897090 (+ 292us) server_negotiation.cc:407] Connection header received
0808 04:05:13.897218 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:05:13.897222 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:05:13.897296 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:05:13.897376 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:05:13.898117 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:13.898918 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:13.900152 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:13.900557 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:13.900643 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:13.904221 (+ 3578us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.1:44839: BlockingRecv error: recv got EOF from 127.5.83.1:44839 (error 108)
Metrics: {"server-negotiator.queue_time_us":127}
Aug 08 04:05:14 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.2@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:05:14 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.2@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:05:14.336170 18651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:14.328207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:40465 (local address 127.5.83.2:54527)
0808 04:05:14.328654 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0808 04:05:14.328675 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 04:05:14.328787 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:05:14.329058 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:05:14.329061 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:05:14.329434 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:05:14.329609 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:14.329619 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:14.330573 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:14.330577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:14.331486 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:14.331498 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:14.332173 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:14.332177 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:14.332292 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:14.332851 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:05:14.332867 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:05:14.335815 (+ 2948us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.62:40465: Server oryx/127.5.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":248,"threads_started":1}
W20260808 04:05:14.336213 18194 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:05:14.328280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.2:54527 (local address 127.5.83.62:40465)
0808 04:05:14.328497 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0808 04:05:14.328503 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:05:14.328750 (+ 247us) server_negotiation.cc:407] Connection header received
0808 04:05:14.328873 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:05:14.328876 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:05:14.328949 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:05:14.329028 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:05:14.329715 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:14.330449 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:14.331595 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:14.332069 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:14.332159 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:14.336040 (+ 3881us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.2:54527: BlockingRecv error: recv got EOF from 127.5.83.2:54527 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Aug 08 04:05:14 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.3@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:05:14 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.3@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:05:14.775386 18652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:14.767281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:40465 (local address 127.5.83.3:58937)
0808 04:05:14.767800 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0808 04:05:14.767819 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 04:05:14.767934 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:05:14.768214 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:05:14.768217 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:05:14.768628 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:05:14.768809 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:14.768820 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:14.769812 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:14.769818 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:14.770740 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:14.770751 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:14.771377 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:14.771390 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:14.771519 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:14.772144 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:05:14.772169 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:05:14.775121 (+ 2952us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.62:40465: Server oryx/127.5.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":261,"threads_started":1}
W20260808 04:05:14.775545 18194 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:05:14.767351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.3:58937 (local address 127.5.83.62:40465)
0808 04:05:14.767521 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0808 04:05:14.767525 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:05:14.767887 (+ 362us) server_negotiation.cc:407] Connection header received
0808 04:05:14.768044 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:05:14.768047 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:05:14.768107 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:05:14.768183 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:05:14.768941 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:14.769668 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:14.770850 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:14.771237 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:14.771321 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:14.775331 (+ 4010us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.3:58937: BlockingRecv error: recv got EOF from 127.5.83.3:58937 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Aug 08 04:05:14 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.1@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:05:14 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.1@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:05:14.913164 18653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:14.905301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:40465 (local address 127.5.83.1:37493)
0808 04:05:14.905765 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0808 04:05:14.905782 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:05:14.905893 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:05:14.906157 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:05:14.906160 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:05:14.906592 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:05:14.906784 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:14.906793 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:14.907754 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:14.907758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:14.908669 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:14.908680 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:14.909342 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:14.909347 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:14.909482 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:14.910015 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:05:14.910031 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:05:14.912917 (+ 2886us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.62:40465: Server oryx/127.5.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":261,"threads_started":1}
W20260808 04:05:14.913313 18194 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:05:14.905358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.1:37493 (local address 127.5.83.62:40465)
0808 04:05:14.905586 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0808 04:05:14.905592 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:05:14.905854 (+ 262us) server_negotiation.cc:407] Connection header received
0808 04:05:14.905978 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:05:14.905981 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:05:14.906057 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:05:14.906132 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:05:14.906901 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:14.907629 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:14.908772 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:14.909224 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:14.909313 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:14.913088 (+ 3775us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.1:37493: BlockingRecv error: recv got EOF from 127.5.83.1:37493 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Aug 08 04:05:15 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.2@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:05:15 dist-test-slave-1l3l krb5kdc[17088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.2@KRBTEST.COM for oryx/127.5.83.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:05:15.345084 18654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:15.337162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:40465 (local address 127.5.83.2:51461)
0808 04:05:15.337670 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0808 04:05:15.337691 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 04:05:15.337800 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:05:15.338060 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:05:15.338063 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:05:15.338476 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:05:15.338673 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:15.338682 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:15.339616 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:15.339619 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:15.340554 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:15.340566 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:15.341216 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:15.341220 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:15.341336 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:15.341876 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:05:15.341892 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:05:15.344830 (+ 2938us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.62:40465: Server oryx/127.5.83.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":278,"threads_started":1}
W20260808 04:05:15.345217 18194 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:05:15.337224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.2:51461 (local address 127.5.83.62:40465)
0808 04:05:15.337372 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0808 04:05:15.337376 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:05:15.337754 (+ 378us) server_negotiation.cc:407] Connection header received
0808 04:05:15.337890 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:05:15.337894 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:05:15.337950 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:05:15.338027 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:05:15.338779 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:15.339508 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:15.340665 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:15.341103 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:15.341199 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:15.345020 (+ 3821us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.2:51461: BlockingRecv error: recv got EOF from 127.5.83.2:51461 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 04:05:15.620186 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 17983
I20260808 04:05:15.632153 18659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:15.621264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50478 (local address 127.5.83.1:36033)
0808 04:05:15.621796 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0808 04:05:15.621804 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:05:15.621831 (+ 27us) server_negotiation.cc:407] Connection header received
0808 04:05:15.621893 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:05:15.621900 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:05:15.622105 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:05:15.622267 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:05:15.623751 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:15.624895 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:15.626170 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:15.626793 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:15.626888 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:15.629350 (+ 2462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:05:15.629358 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:05:15.629372 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:05:15.629418 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:05:15.631666 (+ 2248us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.5.83.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0808 04:05:15.631927 (+ 261us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50478: Request ticket server kudu/127.5.83.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":278,"threads_started":1}
W20260808 04:05:15.632263 18659 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50478: Request ticket server kudu/127.5.83.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260808 04:05:15.632320 5452 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 17983: Not authorized: Client connection negotiation failed: client connection to 127.5.83.1:36033: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.5.83.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260808 04:05:15.632565 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 17983
I20260808 04:05:15.661597 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 18197
I20260808 04:05:15.673151 18661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:15.662629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55864 (local address 127.5.83.2:37893)
0808 04:05:15.663210 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0808 04:05:15.663217 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:05:15.663247 (+ 30us) server_negotiation.cc:407] Connection header received
0808 04:05:15.663311 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:05:15.663318 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:05:15.663536 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:05:15.663747 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:05:15.665056 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:15.666103 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:15.667313 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:15.667890 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:15.668012 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:15.670346 (+ 2334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:05:15.670354 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:05:15.670367 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:05:15.670404 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:05:15.672525 (+ 2121us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.5.83.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0808 04:05:15.672817 (+ 292us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55864: Request ticket server kudu/127.5.83.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":272,"threads_started":1}
W20260808 04:05:15.673210 5452 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 18197: Not authorized: Client connection negotiation failed: client connection to 127.5.83.2:37893: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.5.83.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260808 04:05:15.673249 18661 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55864: Request ticket server kudu/127.5.83.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260808 04:05:15.673341 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 18197
I20260808 04:05:15.690558 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 18383
I20260808 04:05:15.701594 18663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:15.691349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41150 (local address 127.5.83.3:35613)
0808 04:05:15.691963 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0808 04:05:15.691971 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:05:15.691998 (+ 27us) server_negotiation.cc:407] Connection header received
0808 04:05:15.692055 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:05:15.692061 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:05:15.692274 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:05:15.692419 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:05:15.693594 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:15.694654 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:15.695760 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:15.696320 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:15.696426 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:15.698641 (+ 2215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:05:15.698650 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:05:15.698662 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:05:15.698706 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:05:15.701125 (+ 2419us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.5.83.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0808 04:05:15.701352 (+ 227us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41150: Request ticket server kudu/127.5.83.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":307,"threads_started":1}
W20260808 04:05:15.701695 18663 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41150: Request ticket server kudu/127.5.83.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260808 04:05:15.701699 5452 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 18383: Not authorized: Client connection negotiation failed: client connection to 127.5.83.3:35613: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.5.83.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260808 04:05:15.701860 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 18383
I20260808 04:05:15.724990 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 17882
I20260808 04:05:15.726284 17972 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:05:15.843994 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 17882
2026-08-08T04:05:15Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21049 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-08T04:05:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:05:15Z Disabled control of system clock
I20260808 04:05:15.933418 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161826605655-5452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161826605655-5452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161826605655-5452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:38127
--webserver_interface=127.5.83.62
--webserver_port=0
--builtin_ntp_servers=127.5.83.20:33467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.62:38127
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260808 04:05:16.267314 18668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:05:16.267604 18668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:05:16.267690 18668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:05:16.274708 18668 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:05:16.274803 18668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:05:16.274845 18668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:05:16.274900 18668 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:05:16.274942 18668 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:05:16.283779 18668 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:33467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161826605655-5452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161826605655-5452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.62:38127
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:38127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161826605655-5452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:05:16.284986 18668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:05:16.286767 18668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:05:16.297966 18674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:05:16.297937 18676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:05:16.298049 18673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:05:16.298781 18668 server_base.cc:1034] running on GCE node
I20260808 04:05:16.299492 18668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:05:16.300813 18668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:05:16.302023 18668 hybrid_clock.cc:648] HybridClock initialized: now 1786161916301982 us; error 31 us; skew 500 ppm
I20260808 04:05:16.302340 18668 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 04:05:16.304598 18668 webserver.cc:460] Webserver started at http://127.5.83.62:42825/ using document root <none> and password file <none>
I20260808 04:05:16.305233 18668 fs_manager.cc:359] Metadata directory not provided
I20260808 04:05:16.305322 18668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:05:16.305645 18668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:05:16.308455 18668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161826605655-5452-0/minicluster-data/master-0/data/instance:
uuid: "54b0fa2c758d45e5be577b75dfdb4875"
format_stamp: "Formatted at 2026-08-08 04:05:16 on dist-test-slave-1l3l"
server_key: "722e0b8daf09ecac0bef8b56754282f6"
server_key_iv: "95378817e6da5476af1e9b370eb54fd8"
server_key_version: "encryptionkey@0"
I20260808 04:05:16.309222 18668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161826605655-5452-0/minicluster-data/master-0/wal/instance:
uuid: "54b0fa2c758d45e5be577b75dfdb4875"
format_stamp: "Formatted at 2026-08-08 04:05:16 on dist-test-slave-1l3l"
server_key: "722e0b8daf09ecac0bef8b56754282f6"
server_key_iv: "95378817e6da5476af1e9b370eb54fd8"
server_key_version: "encryptionkey@0"
I20260808 04:05:16.314263 18668 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 04:05:16.317646 18687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:05:16.318890 18668 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 04:05:16.319056 18668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161826605655-5452-0/minicluster-data/master-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161826605655-5452-0/minicluster-data/master-0/wal
uuid: "54b0fa2c758d45e5be577b75dfdb4875"
format_stamp: "Formatted at 2026-08-08 04:05:16 on dist-test-slave-1l3l"
server_key: "722e0b8daf09ecac0bef8b56754282f6"
server_key_iv: "95378817e6da5476af1e9b370eb54fd8"
server_key_version: "encryptionkey@0"
I20260808 04:05:16.319236 18668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161826605655-5452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161826605655-5452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161826605655-5452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:05:16.337857 18668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:05:16.338646 18668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:05:16.338908 18668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:05:16.356133 18668 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.62:38127
I20260808 04:05:16.356158 18765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.62:38127 every 8 connection(s)
I20260808 04:05:16.357403 18668 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
I20260808 04:05:16.361055 18766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:05:16.364598 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 18668
I20260808 04:05:16.364804 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161826605655-5452-0/minicluster-data/master-0/wal/instance
I20260808 04:05:16.365693 5452 external_mini_cluster.cc:1428] Setting key 580421a78523c68621c5a17c5f68a8dc
I20260808 04:05:16.373528 18766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54b0fa2c758d45e5be577b75dfdb4875: Bootstrap starting.
I20260808 04:05:16.376552 18766 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54b0fa2c758d45e5be577b75dfdb4875: Neither blocks nor log segments found. Creating new log.
I20260808 04:05:16.377570 18766 log.cc:824] T 00000000000000000000000000000000 P 54b0fa2c758d45e5be577b75dfdb4875: Log is configured to *not* fsync() on all Append() calls
I20260808 04:05:16.378512 18772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:16.367840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60118 (local address 127.5.83.62:38127)
0808 04:05:16.368730 (+ 890us) server_negotiation.cc:206] Beginning negotiation
0808 04:05:16.368740 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 04:05:16.368782 (+ 42us) server_negotiation.cc:407] Connection header received
0808 04:05:16.370304 (+ 1522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:05:16.370325 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:05:16.370785 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:05:16.371143 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:05:16.371740 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:16.372971 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:16.374295 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:16.374990 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:16.375153 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:16.376322 (+ 1169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:05:16.376330 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:05:16.376342 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:05:16.376884 (+ 542us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:05:16.376980 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:05:16.377006 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:05:16.377103 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0808 04:05:16.377324 (+ 221us) server_negotiation.cc:299] Negotiation successful
0808 04:05:16.377601 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":341,"threads_started":1}
I20260808 04:05:16.380401 18766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54b0fa2c758d45e5be577b75dfdb4875: No bootstrap required, opened a new log
I20260808 04:05:16.387509 18766 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54b0fa2c758d45e5be577b75dfdb4875 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54b0fa2c758d45e5be577b75dfdb4875" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 38127 } }
I20260808 04:05:16.387876 18766 raft_consensus.cc:374] T 00000000000000000000000000000000 P 54b0fa2c758d45e5be577b75dfdb4875 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:05:16.387964 18766 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54b0fa2c758d45e5be577b75dfdb4875 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54b0fa2c758d45e5be577b75dfdb4875, State: Initialized, Role: FOLLOWER
I20260808 04:05:16.388451 18766 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54b0fa2c758d45e5be577b75dfdb4875 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54b0fa2c758d45e5be577b75dfdb4875" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 38127 } }
I20260808 04:05:16.388657 18766 raft_consensus.cc:388] T 00000000000000000000000000000000 P 54b0fa2c758d45e5be577b75dfdb4875 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:05:16.388777 18766 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54b0fa2c758d45e5be577b75dfdb4875 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:05:16.388942 18766 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54b0fa2c758d45e5be577b75dfdb4875 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:05:16.391170 18766 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54b0fa2c758d45e5be577b75dfdb4875 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54b0fa2c758d45e5be577b75dfdb4875" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 38127 } }
I20260808 04:05:16.391657 18766 leader_election.cc:302] T 00000000000000000000000000000000 P 54b0fa2c758d45e5be577b75dfdb4875 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 54b0fa2c758d45e5be577b75dfdb4875; no voters:
I20260808 04:05:16.392213 18766 leader_election.cc:288] T 00000000000000000000000000000000 P 54b0fa2c758d45e5be577b75dfdb4875 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:05:16.392331 18775 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 54b0fa2c758d45e5be577b75dfdb4875 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:05:16.392974 18775 raft_consensus.cc:686] T 00000000000000000000000000000000 P 54b0fa2c758d45e5be577b75dfdb4875 [term 1 LEADER]: Becoming Leader. State: Replica: 54b0fa2c758d45e5be577b75dfdb4875, State: Running, Role: LEADER
I20260808 04:05:16.393473 18775 consensus_queue.cc:237] T 00000000000000000000000000000000 P 54b0fa2c758d45e5be577b75dfdb4875 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54b0fa2c758d45e5be577b75dfdb4875" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 38127 } }
I20260808 04:05:16.394102 18766 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54b0fa2c758d45e5be577b75dfdb4875 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:05:16.395982 18780 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54b0fa2c758d45e5be577b75dfdb4875 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "54b0fa2c758d45e5be577b75dfdb4875" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54b0fa2c758d45e5be577b75dfdb4875" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 38127 } } }
I20260808 04:05:16.396477 18780 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54b0fa2c758d45e5be577b75dfdb4875 [sys.catalog]: This master's current role is: LEADER
I20260808 04:05:16.396675 18782 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54b0fa2c758d45e5be577b75dfdb4875 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54b0fa2c758d45e5be577b75dfdb4875. Latest consensus state: current_term: 1 leader_uuid: "54b0fa2c758d45e5be577b75dfdb4875" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54b0fa2c758d45e5be577b75dfdb4875" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 38127 } } }
I20260808 04:05:16.397119 18782 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54b0fa2c758d45e5be577b75dfdb4875 [sys.catalog]: This master's current role is: LEADER
I20260808 04:05:16.401006 18790 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:05:16.407079 18790 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:05:16.414775 18790 catalog_manager.cc:1347] Generated new cluster ID: a852ecc0e1fa458a89ff50d47e574a33
I20260808 04:05:16.414870 18790 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:05:16.429241 18790 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:05:16.430388 18790 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:05:16.435786 18790 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 54b0fa2c758d45e5be577b75dfdb4875: Generated new TSK 0
I20260808 04:05:16.436712 18790 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:05:16.447489 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.1:0
--local_ip_for_outbound_sockets=127.5.83.1
--webserver_interface=127.5.83.1
--webserver_port=0
--tserver_master_addrs=127.5.83.62:38127
--builtin_ntp_servers=127.5.83.20:33467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260808 04:05:16.791685 18805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:05:16.791967 18805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:05:16.792091 18805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:05:16.799146 18805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:05:16.799232 18805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:05:16.799366 18805 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.1
I20260808 04:05:16.808175 18805 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:33467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.83.1
--webserver_port=0
--tserver_master_addrs=127.5.83.62:38127
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.1
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:05:16.809311 18805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:05:16.810977 18805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:05:16.823055 18813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:05:16.823132 18814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:05:16.823051 18817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:05:16.824069 18805 server_base.cc:1034] running on GCE node
I20260808 04:05:16.824630 18805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:05:16.825461 18805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:05:16.826647 18805 hybrid_clock.cc:648] HybridClock initialized: now 1786161916826605 us; error 35 us; skew 500 ppm
I20260808 04:05:16.826929 18805 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 04:05:16.832865 18805 webserver.cc:460] Webserver started at http://127.5.83.1:38725/ using document root <none> and password file <none>
I20260808 04:05:16.833532 18805 fs_manager.cc:359] Metadata directory not provided
I20260808 04:05:16.833630 18805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:05:16.833933 18805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:05:16.836727 18805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-0/data/instance:
uuid: "85c34decfdb749499e819cc789c1256b"
format_stamp: "Formatted at 2026-08-08 04:05:16 on dist-test-slave-1l3l"
server_key: "68c75ea229f5e0b52ec294588877967c"
server_key_iv: "e2b75f25507819871a1be1b87803a225"
server_key_version: "encryptionkey@0"
I20260808 04:05:16.837500 18805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-0/wal/instance:
uuid: "85c34decfdb749499e819cc789c1256b"
format_stamp: "Formatted at 2026-08-08 04:05:16 on dist-test-slave-1l3l"
server_key: "68c75ea229f5e0b52ec294588877967c"
server_key_iv: "e2b75f25507819871a1be1b87803a225"
server_key_version: "encryptionkey@0"
I20260808 04:05:16.842599 18805 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 04:05:16.845899 18825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:05:16.847147 18805 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 04:05:16.847298 18805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-0/wal
uuid: "85c34decfdb749499e819cc789c1256b"
format_stamp: "Formatted at 2026-08-08 04:05:16 on dist-test-slave-1l3l"
server_key: "68c75ea229f5e0b52ec294588877967c"
server_key_iv: "e2b75f25507819871a1be1b87803a225"
server_key_version: "encryptionkey@0"
I20260808 04:05:16.847514 18805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:05:16.881426 18805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:05:16.882197 18805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:05:16.882445 18805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:05:16.883340 18805 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:05:16.884725 18805 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:05:16.884821 18805 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:05:16.884929 18805 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:05:16.885008 18805 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:05:16.914808 18805 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.1:45259
I20260808 04:05:16.914872 18991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.1:45259 every 8 connection(s)
I20260808 04:05:16.916134 18805 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
I20260808 04:05:16.918931 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 18805
I20260808 04:05:16.919117 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-0/wal/instance
I20260808 04:05:16.919977 5452 external_mini_cluster.cc:1428] Setting key 42ed748803dfca9f04e8be72a25dbc56
I20260808 04:05:16.922667 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.2:0
--local_ip_for_outbound_sockets=127.5.83.2
--webserver_interface=127.5.83.2
--webserver_port=0
--tserver_master_addrs=127.5.83.62:38127
--builtin_ntp_servers=127.5.83.20:33467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260808 04:05:16.928561 18995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:16.918605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.1:44915 (local address 127.5.83.62:38127)
0808 04:05:16.919058 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0808 04:05:16.919064 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:05:16.920506 (+ 1442us) server_negotiation.cc:407] Connection header received
0808 04:05:16.922501 (+ 1995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:05:16.922506 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:05:16.922583 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:05:16.922677 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:05:16.923771 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:16.924512 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:16.925978 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:16.926376 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:16.926482 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:16.927760 (+ 1278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:05:16.927766 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:05:16.927771 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:05:16.927790 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:05:16.927837 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:05:16.927849 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:05:16.927924 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 04:05:16.928124 (+ 200us) server_negotiation.cc:299] Negotiation successful
0808 04:05:16.928214 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":250,"threads_started":1}
I20260808 04:05:16.929461 18996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:16.919306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:38127 (local address 127.5.83.1:44915)
0808 04:05:16.920311 (+ 1005us) negotiation.cc:107] Waiting for socket to connect
0808 04:05:16.920385 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0808 04:05:16.922250 (+ 1865us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:05:16.922920 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:05:16.922931 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:05:16.923020 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:05:16.923598 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:16.923615 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:16.924646 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:16.924650 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:16.925841 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:16.925853 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:16.926581 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:16.926588 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:16.926802 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:16.927466 (+ 664us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:05:16.927468 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:05:16.927504 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:05:16.927571 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:05:16.927935 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:05:16.927946 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:05:16.927953 (+ 7us) client_negotiation.cc:769] Sending connection context
0808 04:05:16.928101 (+ 148us) client_negotiation.cc:240] Negotiation successful
0808 04:05:16.928467 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":350,"threads_started":1}
I20260808 04:05:16.930846 18993 heartbeater.cc:347] Connected to a master server at 127.5.83.62:38127
I20260808 04:05:16.931113 18993 heartbeater.cc:464] Registering TS with master...
I20260808 04:05:16.931901 18993 heartbeater.cc:511] Master 127.5.83.62:38127 requested a full tablet report, sending...
I20260808 04:05:16.933805 18715 ts_manager.cc:194] Registered new tserver with Master: 85c34decfdb749499e819cc789c1256b (127.5.83.1:45259)
I20260808 04:05:16.935256 18715 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.83.1:44915
W20260808 04:05:17.245260 18998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:05:17.245601 18998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:05:17.245728 18998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:05:17.253168 18998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:05:17.253278 18998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:05:17.253425 18998 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.2
I20260808 04:05:17.262995 18998 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:33467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.83.2
--webserver_port=0
--tserver_master_addrs=127.5.83.62:38127
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.2
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:05:17.264336 18998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:05:17.266115 18998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:05:17.278707 19006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:05:17.278731 19009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:05:17.279134 19007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:05:17.279680 18998 server_base.cc:1034] running on GCE node
I20260808 04:05:17.280287 18998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:05:17.281286 18998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:05:17.282524 18998 hybrid_clock.cc:648] HybridClock initialized: now 1786161917282471 us; error 38 us; skew 500 ppm
I20260808 04:05:17.282872 18998 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 04:05:17.285318 18998 webserver.cc:460] Webserver started at http://127.5.83.2:34627/ using document root <none> and password file <none>
I20260808 04:05:17.286078 18998 fs_manager.cc:359] Metadata directory not provided
I20260808 04:05:17.286175 18998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:05:17.286509 18998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:05:17.289368 18998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-1/data/instance:
uuid: "566f2da26ef94620b226c8ed304f776e"
format_stamp: "Formatted at 2026-08-08 04:05:17 on dist-test-slave-1l3l"
server_key: "e4097a5091bcb7224f4a21b676b20180"
server_key_iv: "4c748fa515f60e64322d120736fe25dc"
server_key_version: "encryptionkey@0"
I20260808 04:05:17.290177 18998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-1/wal/instance:
uuid: "566f2da26ef94620b226c8ed304f776e"
format_stamp: "Formatted at 2026-08-08 04:05:17 on dist-test-slave-1l3l"
server_key: "e4097a5091bcb7224f4a21b676b20180"
server_key_iv: "4c748fa515f60e64322d120736fe25dc"
server_key_version: "encryptionkey@0"
I20260808 04:05:17.295251 18998 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260808 04:05:17.298704 19015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:05:17.300158 18998 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:05:17.300318 18998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-1/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-1/wal
uuid: "566f2da26ef94620b226c8ed304f776e"
format_stamp: "Formatted at 2026-08-08 04:05:17 on dist-test-slave-1l3l"
server_key: "e4097a5091bcb7224f4a21b676b20180"
server_key_iv: "4c748fa515f60e64322d120736fe25dc"
server_key_version: "encryptionkey@0"
I20260808 04:05:17.300537 18998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:05:17.324007 18998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:05:17.324776 18998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:05:17.325030 18998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:05:17.325922 18998 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:05:17.327337 18998 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:05:17.327430 18998 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:05:17.327535 18998 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:05:17.327613 18998 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:05:17.359038 18998 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.2:37185
I20260808 04:05:17.359098 19177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.2:37185 every 8 connection(s)
I20260808 04:05:17.360486 18998 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
I20260808 04:05:17.368563 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 18998
I20260808 04:05:17.368728 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-1/wal/instance
I20260808 04:05:17.369264 5452 external_mini_cluster.cc:1428] Setting key ce23507abb969d0865600b9c5c982baa
I20260808 04:05:17.371910 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.3:0
--local_ip_for_outbound_sockets=127.5.83.3
--webserver_interface=127.5.83.3
--webserver_port=0
--tserver_master_addrs=127.5.83.62:38127
--builtin_ntp_servers=127.5.83.20:33467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260808 04:05:17.373170 18995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:17.363075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.2:54837 (local address 127.5.83.62:38127)
0808 04:05:17.363235 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0808 04:05:17.363239 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:05:17.364705 (+ 1466us) server_negotiation.cc:407] Connection header received
0808 04:05:17.366428 (+ 1723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:05:17.366434 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:05:17.366513 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:05:17.366600 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:05:17.367599 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:17.368395 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:17.370076 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:17.370721 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:17.370835 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:17.372464 (+ 1629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:05:17.372470 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:05:17.372475 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:05:17.372495 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:05:17.372531 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:05:17.372545 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:05:17.372633 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0808 04:05:17.372791 (+ 158us) server_negotiation.cc:299] Negotiation successful
0808 04:05:17.372894 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 04:05:17.374015 19182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:17.363639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:38127 (local address 127.5.83.2:54837)
0808 04:05:17.364549 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0808 04:05:17.364588 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 04:05:17.366180 (+ 1592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:05:17.366815 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:05:17.366823 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:05:17.366915 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:05:17.367439 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:17.367454 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:17.368546 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:17.368550 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:17.369905 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:17.369922 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:17.371077 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:17.371084 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:17.371370 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:17.372182 (+ 812us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:05:17.372185 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:05:17.372220 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:05:17.372292 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:05:17.372629 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:05:17.372634 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:05:17.372640 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 04:05:17.372764 (+ 124us) client_negotiation.cc:240] Negotiation successful
0808 04:05:17.373027 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":284,"threads_started":1}
I20260808 04:05:17.375172 19180 heartbeater.cc:347] Connected to a master server at 127.5.83.62:38127
I20260808 04:05:17.375483 19180 heartbeater.cc:464] Registering TS with master...
I20260808 04:05:17.376327 19180 heartbeater.cc:511] Master 127.5.83.62:38127 requested a full tablet report, sending...
I20260808 04:05:17.377727 18715 ts_manager.cc:194] Registered new tserver with Master: 566f2da26ef94620b226c8ed304f776e (127.5.83.2:37185)
I20260808 04:05:17.378515 18715 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.83.2:54837
W20260808 04:05:17.692209 19184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:05:17.692485 19184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:05:17.692605 19184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:05:17.699829 19184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:05:17.699920 19184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:05:17.700050 19184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.3
I20260808 04:05:17.708859 19184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:33467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.83.3
--webserver_port=0
--tserver_master_addrs=127.5.83.62:38127
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.3
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:05:17.710008 19184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:05:17.711665 19184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:05:17.724036 19200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:05:17.723994 19202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:05:17.724064 19205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:05:17.724954 19184 server_base.cc:1034] running on GCE node
I20260808 04:05:17.725500 19184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:05:17.726432 19184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:05:17.727619 19184 hybrid_clock.cc:648] HybridClock initialized: now 1786161917727575 us; error 37 us; skew 500 ppm
I20260808 04:05:17.727921 19184 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 04:05:17.729945 19184 webserver.cc:460] Webserver started at http://127.5.83.3:35063/ using document root <none> and password file <none>
I20260808 04:05:17.730629 19184 fs_manager.cc:359] Metadata directory not provided
I20260808 04:05:17.730733 19184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:05:17.731042 19184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:05:17.733736 19184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-2/data/instance:
uuid: "82dd7886fbd84923adce7c7d3ddf581b"
format_stamp: "Formatted at 2026-08-08 04:05:17 on dist-test-slave-1l3l"
server_key: "94a1e074c2545ff7adf267aff4edc98f"
server_key_iv: "1e1695ba030edc20c1d818c4a02a6486"
server_key_version: "encryptionkey@0"
I20260808 04:05:17.734494 19184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-2/wal/instance:
uuid: "82dd7886fbd84923adce7c7d3ddf581b"
format_stamp: "Formatted at 2026-08-08 04:05:17 on dist-test-slave-1l3l"
server_key: "94a1e074c2545ff7adf267aff4edc98f"
server_key_iv: "1e1695ba030edc20c1d818c4a02a6486"
server_key_version: "encryptionkey@0"
I20260808 04:05:17.739238 19184 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260808 04:05:17.742313 19214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:05:17.743481 19184 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260808 04:05:17.743628 19184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-2/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-2/wal
uuid: "82dd7886fbd84923adce7c7d3ddf581b"
format_stamp: "Formatted at 2026-08-08 04:05:17 on dist-test-slave-1l3l"
server_key: "94a1e074c2545ff7adf267aff4edc98f"
server_key_iv: "1e1695ba030edc20c1d818c4a02a6486"
server_key_version: "encryptionkey@0"
I20260808 04:05:17.743867 19184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:05:17.754329 19184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:05:17.755015 19184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:05:17.755254 19184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:05:17.756135 19184 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:05:17.757494 19184 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:05:17.757596 19184 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:05:17.757700 19184 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:05:17.757777 19184 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:05:17.787768 19184 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.3:35167
I20260808 04:05:17.787825 19383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.3:35167 every 8 connection(s)
I20260808 04:05:17.789029 19184 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
I20260808 04:05:17.797246 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 19184
I20260808 04:05:17.797423 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-2/wal/instance
I20260808 04:05:17.797971 5452 external_mini_cluster.cc:1428] Setting key be8bca5ee87e75dd87d84d85dec7e3a5
I20260808 04:05:17.800789 18995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:17.791329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.3:43827 (local address 127.5.83.62:38127)
0808 04:05:17.791489 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0808 04:05:17.791493 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:05:17.792764 (+ 1271us) server_negotiation.cc:407] Connection header received
0808 04:05:17.794396 (+ 1632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:05:17.794401 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:05:17.794485 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:05:17.794567 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:05:17.795520 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:17.796293 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:17.798245 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:17.798711 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:17.798837 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:17.800127 (+ 1290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:05:17.800132 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:05:17.800138 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:05:17.800155 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:05:17.800186 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:05:17.800196 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:05:17.800261 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0808 04:05:17.800466 (+ 205us) server_negotiation.cc:299] Negotiation successful
0808 04:05:17.800539 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 04:05:17.801842 19388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:17.791772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:38127 (local address 127.5.83.3:43827)
0808 04:05:17.792613 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0808 04:05:17.792654 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 04:05:17.794161 (+ 1507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:05:17.794738 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:05:17.794746 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:05:17.794827 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:05:17.795370 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:17.795386 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:17.796413 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:17.796417 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:17.798092 (+ 1675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:17.798106 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:17.798917 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:17.798922 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:17.799207 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:17.799857 (+ 650us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:05:17.799859 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:05:17.799895 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:05:17.799973 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:05:17.800282 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:05:17.800292 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:05:17.800300 (+ 8us) client_negotiation.cc:769] Sending connection context
0808 04:05:17.800471 (+ 171us) client_negotiation.cc:240] Negotiation successful
0808 04:05:17.800824 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":259,"threads_started":1}
I20260808 04:05:17.803009 19384 heartbeater.cc:347] Connected to a master server at 127.5.83.62:38127
I20260808 04:05:17.803287 19384 heartbeater.cc:464] Registering TS with master...
I20260808 04:05:17.804093 19384 heartbeater.cc:511] Master 127.5.83.62:38127 requested a full tablet report, sending...
I20260808 04:05:17.805440 18715 ts_manager.cc:194] Registered new tserver with Master: 82dd7886fbd84923adce7c7d3ddf581b (127.5.83.3:35167)
I20260808 04:05:17.806170 18715 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.83.3:43827
I20260808 04:05:17.813212 5452 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260808 04:05:17.818856 18995 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:05:17.817361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60122 (local address 127.5.83.62:38127)
0808 04:05:17.817576 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0808 04:05:17.817583 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:05:17.817946 (+ 363us) server_negotiation.cc:407] Connection header received
0808 04:05:17.818115 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:05:17.818121 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:05:17.818207 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:05:17.818277 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:05:17.818669 (+ 392us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60122: BlockingRecv error: recv got EOF from 127.0.0.1:60122 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 04:05:17.819553 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 18805
I20260808 04:05:17.826565 19399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:17.820477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56490 (local address 127.5.83.1:45259)
0808 04:05:17.820985 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0808 04:05:17.820990 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:05:17.821013 (+ 23us) server_negotiation.cc:407] Connection header received
0808 04:05:17.821071 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:05:17.821076 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:05:17.821233 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:05:17.821350 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:05:17.821609 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:17.822627 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:17.823898 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:17.824404 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:17.824507 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:17.825348 (+ 841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:05:17.825356 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:05:17.825367 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:05:17.825891 (+ 524us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:05:17.825948 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:05:17.825967 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:05:17.826050 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0808 04:05:17.826127 (+ 77us) server_negotiation.cc:299] Negotiation successful
0808 04:05:17.826231 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":258,"threads_started":1}
I20260808 04:05:17.827641 18987 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:05:17.907392 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 18805
I20260808 04:05:17.930478 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 18998
I20260808 04:05:17.937709 19404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:17.931338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54542 (local address 127.5.83.2:37185)
0808 04:05:17.931930 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0808 04:05:17.931937 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:05:17.931964 (+ 27us) server_negotiation.cc:407] Connection header received
0808 04:05:17.932027 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:05:17.932034 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:05:17.932244 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:05:17.932366 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:05:17.932755 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:17.933754 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:17.934879 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:17.935432 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:17.935526 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:17.936454 (+ 928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:05:17.936462 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:05:17.936475 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:05:17.937056 (+ 581us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:05:17.937103 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:05:17.937122 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:05:17.937206 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0808 04:05:17.937312 (+ 106us) server_negotiation.cc:299] Negotiation successful
0808 04:05:17.937420 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":278,"threads_started":1}
I20260808 04:05:17.938871 19170 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:05:18.019168 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 18998
I20260808 04:05:18.037773 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 19184
I20260808 04:05:18.045269 19407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:18.038616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34152 (local address 127.5.83.3:35167)
0808 04:05:18.039235 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0808 04:05:18.039241 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:05:18.039279 (+ 38us) server_negotiation.cc:407] Connection header received
0808 04:05:18.039339 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:05:18.039345 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:05:18.039555 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:05:18.039680 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:05:18.040079 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:18.041108 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:18.042307 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:18.042854 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:18.042956 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:18.044006 (+ 1050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:05:18.044015 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:05:18.044026 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:05:18.044598 (+ 572us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:05:18.044643 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:05:18.044662 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:05:18.044747 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0808 04:05:18.044855 (+ 108us) server_negotiation.cc:299] Negotiation successful
0808 04:05:18.044977 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":309,"threads_started":1}
I20260808 04:05:18.046478 19379 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:05:18.125694 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 19184
I20260808 04:05:18.144982 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 18668
I20260808 04:05:18.146212 18760 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:05:18.246954 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 18668
2026-08-08T04:05:18Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2396 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-08T04:05:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:05:18Z Disabled control of system clock
I20260808 04:05:18.324301 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161826605655-5452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161826605655-5452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161826605655-5452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:38033
--webserver_interface=127.5.83.62
--webserver_port=0
--builtin_ntp_servers=127.5.83.20:39473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.62:38033
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260808 04:05:18.635057 19416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:05:18.635344 19416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:05:18.635440 19416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:05:18.642765 19416 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:05:18.642860 19416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:05:18.642904 19416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:05:18.642958 19416 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:05:18.643000 19416 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:05:18.651748 19416 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:39473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161826605655-5452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161826605655-5452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.62:38033
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:38033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161826605655-5452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:05:18.652886 19416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:05:18.654565 19416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:05:18.665306 19423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:05:18.665342 19425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:05:18.665524 19428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:05:18.666148 19416 server_base.cc:1034] running on GCE node
I20260808 04:05:18.666795 19416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:05:18.668013 19416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:05:18.669198 19416 hybrid_clock.cc:648] HybridClock initialized: now 1786161918669145 us; error 45 us; skew 500 ppm
I20260808 04:05:18.669481 19416 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 04:05:18.671545 19416 webserver.cc:460] Webserver started at http://127.5.83.62:38521/ using document root <none> and password file <none>
I20260808 04:05:18.672181 19416 fs_manager.cc:359] Metadata directory not provided
I20260808 04:05:18.672266 19416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:05:18.672523 19416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:05:18.675158 19416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161826605655-5452-0/minicluster-data/master-0/data/instance:
uuid: "ad6b4cb0639b4dabac107328af784a58"
format_stamp: "Formatted at 2026-08-08 04:05:18 on dist-test-slave-1l3l"
server_key: "435cf0e4cd3f4ed9cb69983e27b79de8"
server_key_iv: "6270b36dafa2a05fad27bcd594cef805"
server_key_version: "encryptionkey@0"
I20260808 04:05:18.675937 19416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161826605655-5452-0/minicluster-data/master-0/wal/instance:
uuid: "ad6b4cb0639b4dabac107328af784a58"
format_stamp: "Formatted at 2026-08-08 04:05:18 on dist-test-slave-1l3l"
server_key: "435cf0e4cd3f4ed9cb69983e27b79de8"
server_key_iv: "6270b36dafa2a05fad27bcd594cef805"
server_key_version: "encryptionkey@0"
I20260808 04:05:18.680572 19416 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260808 04:05:18.683583 19436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:05:18.684756 19416 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 04:05:18.684908 19416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161826605655-5452-0/minicluster-data/master-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161826605655-5452-0/minicluster-data/master-0/wal
uuid: "ad6b4cb0639b4dabac107328af784a58"
format_stamp: "Formatted at 2026-08-08 04:05:18 on dist-test-slave-1l3l"
server_key: "435cf0e4cd3f4ed9cb69983e27b79de8"
server_key_iv: "6270b36dafa2a05fad27bcd594cef805"
server_key_version: "encryptionkey@0"
I20260808 04:05:18.685091 19416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161826605655-5452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161826605655-5452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161826605655-5452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:05:18.690405 19416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:05:18.691097 19416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:05:18.691352 19416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:05:18.707652 19416 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.62:38033
I20260808 04:05:18.707690 19505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.62:38033 every 8 connection(s)
I20260808 04:05:18.708843 19416 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
I20260808 04:05:18.712282 19506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:05:18.712688 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 19416
I20260808 04:05:18.712880 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161826605655-5452-0/minicluster-data/master-0/wal/instance
I20260808 04:05:18.713557 5452 external_mini_cluster.cc:1428] Setting key 6976dacee71564f3e143b2140d9db7c2
I20260808 04:05:18.721210 19512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:18.715424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33102 (local address 127.5.83.62:38033)
0808 04:05:18.716403 (+ 979us) server_negotiation.cc:206] Beginning negotiation
0808 04:05:18.716413 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 04:05:18.716452 (+ 39us) server_negotiation.cc:407] Connection header received
0808 04:05:18.717953 (+ 1501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:05:18.717972 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:05:18.718348 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:05:18.718652 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:05:18.719749 (+ 1097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:05:18.719757 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:05:18.719772 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:05:18.719788 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:05:18.719848 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:05:18.719864 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:05:18.720041 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0808 04:05:18.720141 (+ 100us) server_negotiation.cc:299] Negotiation successful
0808 04:05:18.720355 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":353,"threads_started":1}
I20260808 04:05:18.724323 19506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad6b4cb0639b4dabac107328af784a58: Bootstrap starting.
I20260808 04:05:18.727001 19506 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad6b4cb0639b4dabac107328af784a58: Neither blocks nor log segments found. Creating new log.
I20260808 04:05:18.727991 19506 log.cc:824] T 00000000000000000000000000000000 P ad6b4cb0639b4dabac107328af784a58: Log is configured to *not* fsync() on all Append() calls
I20260808 04:05:18.730628 19506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad6b4cb0639b4dabac107328af784a58: No bootstrap required, opened a new log
I20260808 04:05:18.737569 19506 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad6b4cb0639b4dabac107328af784a58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad6b4cb0639b4dabac107328af784a58" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 38033 } }
I20260808 04:05:18.737875 19506 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad6b4cb0639b4dabac107328af784a58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:05:18.737978 19506 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad6b4cb0639b4dabac107328af784a58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad6b4cb0639b4dabac107328af784a58, State: Initialized, Role: FOLLOWER
I20260808 04:05:18.738472 19506 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad6b4cb0639b4dabac107328af784a58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad6b4cb0639b4dabac107328af784a58" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 38033 } }
I20260808 04:05:18.738673 19506 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad6b4cb0639b4dabac107328af784a58 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:05:18.738833 19506 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad6b4cb0639b4dabac107328af784a58 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:05:18.739012 19506 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad6b4cb0639b4dabac107328af784a58 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:05:18.741065 19506 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad6b4cb0639b4dabac107328af784a58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad6b4cb0639b4dabac107328af784a58" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 38033 } }
I20260808 04:05:18.741497 19506 leader_election.cc:302] T 00000000000000000000000000000000 P ad6b4cb0639b4dabac107328af784a58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad6b4cb0639b4dabac107328af784a58; no voters:
I20260808 04:05:18.742038 19506 leader_election.cc:288] T 00000000000000000000000000000000 P ad6b4cb0639b4dabac107328af784a58 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:05:18.742147 19514 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad6b4cb0639b4dabac107328af784a58 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:05:18.742924 19514 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad6b4cb0639b4dabac107328af784a58 [term 1 LEADER]: Becoming Leader. State: Replica: ad6b4cb0639b4dabac107328af784a58, State: Running, Role: LEADER
I20260808 04:05:18.743410 19514 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad6b4cb0639b4dabac107328af784a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad6b4cb0639b4dabac107328af784a58" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 38033 } }
I20260808 04:05:18.744127 19506 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad6b4cb0639b4dabac107328af784a58 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:05:18.746140 19515 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad6b4cb0639b4dabac107328af784a58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad6b4cb0639b4dabac107328af784a58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad6b4cb0639b4dabac107328af784a58" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 38033 } } }
I20260808 04:05:18.746166 19516 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad6b4cb0639b4dabac107328af784a58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad6b4cb0639b4dabac107328af784a58. Latest consensus state: current_term: 1 leader_uuid: "ad6b4cb0639b4dabac107328af784a58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad6b4cb0639b4dabac107328af784a58" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 38033 } } }
I20260808 04:05:18.746630 19515 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad6b4cb0639b4dabac107328af784a58 [sys.catalog]: This master's current role is: LEADER
I20260808 04:05:18.746668 19516 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad6b4cb0639b4dabac107328af784a58 [sys.catalog]: This master's current role is: LEADER
I20260808 04:05:18.748180 19523 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:05:18.753207 19523 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:05:18.761744 19523 catalog_manager.cc:1347] Generated new cluster ID: fb6715c163064fa8bb4d35b4191b8b65
I20260808 04:05:18.761834 19523 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:05:18.797541 19523 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:05:18.798012 19523 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:05:18.809746 19523 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad6b4cb0639b4dabac107328af784a58: Generated new TSK 0
I20260808 04:05:18.810683 19523 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:05:18.817059 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.1:0
--local_ip_for_outbound_sockets=127.5.83.1
--webserver_interface=127.5.83.1
--webserver_port=0
--tserver_master_addrs=127.5.83.62:38033
--builtin_ntp_servers=127.5.83.20:39473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260808 04:05:19.144333 19548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:05:19.144596 19548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:05:19.144723 19548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:05:19.151842 19548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:05:19.151935 19548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:05:19.152072 19548 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.1
I20260808 04:05:19.160667 19548 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:39473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.83.1
--webserver_port=0
--tserver_master_addrs=127.5.83.62:38033
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.1
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:05:19.161820 19548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:05:19.163506 19548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:05:19.175899 19559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:05:19.176033 19562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:05:19.175940 19560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:05:19.176710 19548 server_base.cc:1034] running on GCE node
I20260808 04:05:19.177270 19548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:05:19.178165 19548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:05:19.179378 19548 hybrid_clock.cc:648] HybridClock initialized: now 1786161919179313 us; error 47 us; skew 500 ppm
I20260808 04:05:19.179674 19548 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 04:05:19.181824 19548 webserver.cc:460] Webserver started at http://127.5.83.1:41285/ using document root <none> and password file <none>
I20260808 04:05:19.182478 19548 fs_manager.cc:359] Metadata directory not provided
I20260808 04:05:19.182595 19548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:05:19.182891 19548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:05:19.185616 19548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-0/data/instance:
uuid: "110f0337915f43e58081691631f56f23"
format_stamp: "Formatted at 2026-08-08 04:05:19 on dist-test-slave-1l3l"
server_key: "4bb7cc9ab865b374fcf02736c06ec7f9"
server_key_iv: "7521716e58e75baa0694406aa1dbf86e"
server_key_version: "encryptionkey@0"
I20260808 04:05:19.186461 19548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-0/wal/instance:
uuid: "110f0337915f43e58081691631f56f23"
format_stamp: "Formatted at 2026-08-08 04:05:19 on dist-test-slave-1l3l"
server_key: "4bb7cc9ab865b374fcf02736c06ec7f9"
server_key_iv: "7521716e58e75baa0694406aa1dbf86e"
server_key_version: "encryptionkey@0"
I20260808 04:05:19.191466 19548 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260808 04:05:19.194761 19578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:05:19.196039 19548 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:05:19.196183 19548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-0/wal
uuid: "110f0337915f43e58081691631f56f23"
format_stamp: "Formatted at 2026-08-08 04:05:19 on dist-test-slave-1l3l"
server_key: "4bb7cc9ab865b374fcf02736c06ec7f9"
server_key_iv: "7521716e58e75baa0694406aa1dbf86e"
server_key_version: "encryptionkey@0"
I20260808 04:05:19.196372 19548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:05:19.201467 19548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:05:19.202188 19548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:05:19.202421 19548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:05:19.203295 19548 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:05:19.204614 19548 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:05:19.204706 19548 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:05:19.204815 19548 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:05:19.204888 19548 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:05:19.234602 19548 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.1:44783
I20260808 04:05:19.234653 19757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.1:44783 every 8 connection(s)
I20260808 04:05:19.235939 19548 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
I20260808 04:05:19.243582 19761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:19.238308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.1:35915 (local address 127.5.83.62:38033)
0808 04:05:19.238731 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0808 04:05:19.238738 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:05:19.239891 (+ 1153us) server_negotiation.cc:407] Connection header received
0808 04:05:19.241547 (+ 1656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:05:19.241551 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:05:19.241604 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:05:19.241699 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:05:19.242914 (+ 1215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:05:19.242918 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:05:19.242921 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:05:19.242937 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:05:19.242981 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:05:19.242993 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:05:19.243094 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0808 04:05:19.243231 (+ 137us) server_negotiation.cc:299] Negotiation successful
0808 04:05:19.243283 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":196,"threads_started":1}
I20260808 04:05:19.243834 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 19548
I20260808 04:05:19.244035 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-0/wal/instance
I20260808 04:05:19.244539 19764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:19.238840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:38033 (local address 127.5.83.1:35915)
0808 04:05:19.239698 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0808 04:05:19.239786 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0808 04:05:19.241307 (+ 1521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:05:19.241873 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:05:19.241895 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:05:19.241978 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:05:19.242608 (+ 630us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:05:19.242610 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:05:19.242647 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:05:19.242730 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:05:19.243074 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:05:19.243079 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:05:19.243084 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 04:05:19.243310 (+ 226us) client_negotiation.cc:240] Negotiation successful
0808 04:05:19.243491 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":269,"threads_started":1}
I20260808 04:05:19.244746 5452 external_mini_cluster.cc:1428] Setting key 619de6b0924f995ed6da0d1cea44edd3
I20260808 04:05:19.245982 19758 heartbeater.cc:347] Connected to a master server at 127.5.83.62:38033
I20260808 04:05:19.246238 19758 heartbeater.cc:464] Registering TS with master...
I20260808 04:05:19.246852 19758 heartbeater.cc:511] Master 127.5.83.62:38033 requested a full tablet report, sending...
I20260808 04:05:19.247678 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.2:0
--local_ip_for_outbound_sockets=127.5.83.2
--webserver_interface=127.5.83.2
--webserver_port=0
--tserver_master_addrs=127.5.83.62:38033
--builtin_ntp_servers=127.5.83.20:39473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260808 04:05:19.248799 19458 ts_manager.cc:194] Registered new tserver with Master: 110f0337915f43e58081691631f56f23 (127.5.83.1:44783)
W20260808 04:05:19.574270 19766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:05:19.574512 19766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:05:19.574595 19766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:05:19.581561 19766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:05:19.581650 19766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:05:19.581758 19766 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.2
I20260808 04:05:19.590560 19766 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:39473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.83.2
--webserver_port=0
--tserver_master_addrs=127.5.83.62:38033
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.2
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:05:19.591782 19766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:05:19.593480 19766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:05:19.605441 19773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:05:19.605485 19774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:05:19.605520 19777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:05:19.606616 19766 server_base.cc:1034] running on GCE node
I20260808 04:05:19.607143 19766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:05:19.608031 19766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:05:19.609226 19766 hybrid_clock.cc:648] HybridClock initialized: now 1786161919609192 us; error 28 us; skew 500 ppm
I20260808 04:05:19.609519 19766 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 04:05:19.611553 19766 webserver.cc:460] Webserver started at http://127.5.83.2:40987/ using document root <none> and password file <none>
I20260808 04:05:19.612205 19766 fs_manager.cc:359] Metadata directory not provided
I20260808 04:05:19.612305 19766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:05:19.612632 19766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:05:19.615324 19766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-1/data/instance:
uuid: "13c3ddab6f4f4dea9b20b4140bd35ee1"
format_stamp: "Formatted at 2026-08-08 04:05:19 on dist-test-slave-1l3l"
server_key: "ce408c18ca800d635d7efc15a6dbed86"
server_key_iv: "1183a199ee0c76f62dbbb881d1ccb504"
server_key_version: "encryptionkey@0"
I20260808 04:05:19.616122 19766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-1/wal/instance:
uuid: "13c3ddab6f4f4dea9b20b4140bd35ee1"
format_stamp: "Formatted at 2026-08-08 04:05:19 on dist-test-slave-1l3l"
server_key: "ce408c18ca800d635d7efc15a6dbed86"
server_key_iv: "1183a199ee0c76f62dbbb881d1ccb504"
server_key_version: "encryptionkey@0"
I20260808 04:05:19.620852 19766 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260808 04:05:19.624258 19788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:05:19.625591 19766 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:05:19.625746 19766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-1/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-1/wal
uuid: "13c3ddab6f4f4dea9b20b4140bd35ee1"
format_stamp: "Formatted at 2026-08-08 04:05:19 on dist-test-slave-1l3l"
server_key: "ce408c18ca800d635d7efc15a6dbed86"
server_key_iv: "1183a199ee0c76f62dbbb881d1ccb504"
server_key_version: "encryptionkey@0"
I20260808 04:05:19.625958 19766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:05:19.631526 19766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:05:19.632335 19766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:05:19.632594 19766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:05:19.633474 19766 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:05:19.634795 19766 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:05:19.634891 19766 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:05:19.634997 19766 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:05:19.635072 19766 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:05:19.664609 19766 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.2:45093
I20260808 04:05:19.664669 19961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.2:45093 every 8 connection(s)
I20260808 04:05:19.665866 19766 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
I20260808 04:05:19.672580 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 19766
I20260808 04:05:19.672756 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-1/wal/instance
I20260808 04:05:19.673182 5452 external_mini_cluster.cc:1428] Setting key e46aa632e0aa27497754d63f8cf1c7ac
I20260808 04:05:19.673491 19761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:19.668229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.2:57051 (local address 127.5.83.62:38033)
0808 04:05:19.668406 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0808 04:05:19.668411 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:05:19.669713 (+ 1302us) server_negotiation.cc:407] Connection header received
0808 04:05:19.671327 (+ 1614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:05:19.671332 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:05:19.671415 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:05:19.671505 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:05:19.672777 (+ 1272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:05:19.672782 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:05:19.672785 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:05:19.672800 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:05:19.672833 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:05:19.672847 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:05:19.672957 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0808 04:05:19.673128 (+ 171us) server_negotiation.cc:299] Negotiation successful
0808 04:05:19.673195 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 04:05:19.674444 19968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:19.668715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:38033 (local address 127.5.83.2:57051)
0808 04:05:19.669575 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0808 04:05:19.669609 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0808 04:05:19.671080 (+ 1471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:05:19.671752 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:05:19.671762 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:05:19.671849 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:05:19.672476 (+ 627us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:05:19.672478 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:05:19.672514 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:05:19.672595 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:05:19.672930 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:05:19.672942 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:05:19.672950 (+ 8us) client_negotiation.cc:769] Sending connection context
0808 04:05:19.673212 (+ 262us) client_negotiation.cc:240] Negotiation successful
0808 04:05:19.673437 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":273,"threads_started":1}
I20260808 04:05:19.675899 19962 heartbeater.cc:347] Connected to a master server at 127.5.83.62:38033
I20260808 04:05:19.676182 19962 heartbeater.cc:464] Registering TS with master...
I20260808 04:05:19.676182 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.3:0
--local_ip_for_outbound_sockets=127.5.83.3
--webserver_interface=127.5.83.3
--webserver_port=0
--tserver_master_addrs=127.5.83.62:38033
--builtin_ntp_servers=127.5.83.20:39473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260808 04:05:19.676743 19962 heartbeater.cc:511] Master 127.5.83.62:38033 requested a full tablet report, sending...
I20260808 04:05:19.678120 19458 ts_manager.cc:194] Registered new tserver with Master: 13c3ddab6f4f4dea9b20b4140bd35ee1 (127.5.83.2:45093)
W20260808 04:05:19.991044 19969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:05:19.991312 19969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:05:19.991452 19969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:05:19.998391 19969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:05:19.998492 19969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:05:19.998637 19969 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.3
I20260808 04:05:20.007452 19969 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:39473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.83.3
--webserver_port=0
--tserver_master_addrs=127.5.83.62:38033
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.3
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:05:20.008665 19969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:05:20.010432 19969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:05:20.022771 19983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:05:20.022816 19978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:05:20.022846 19976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:05:20.023618 19969 server_base.cc:1034] running on GCE node
I20260808 04:05:20.024240 19969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:05:20.025106 19969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:05:20.026289 19969 hybrid_clock.cc:648] HybridClock initialized: now 1786161920026223 us; error 62 us; skew 500 ppm
I20260808 04:05:20.026573 19969 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 04:05:20.028705 19969 webserver.cc:460] Webserver started at http://127.5.83.3:43403/ using document root <none> and password file <none>
I20260808 04:05:20.029352 19969 fs_manager.cc:359] Metadata directory not provided
I20260808 04:05:20.029454 19969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:05:20.029795 19969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:05:20.032537 19969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-2/data/instance:
uuid: "724c8b24bc224ae58e2bb9d1f6d26c0e"
format_stamp: "Formatted at 2026-08-08 04:05:20 on dist-test-slave-1l3l"
server_key: "1ce4bdf903f75c6c895095095dc4d0e5"
server_key_iv: "1bdc7d30843e156a376fc71c6423e3ac"
server_key_version: "encryptionkey@0"
I20260808 04:05:20.033244 19969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-2/wal/instance:
uuid: "724c8b24bc224ae58e2bb9d1f6d26c0e"
format_stamp: "Formatted at 2026-08-08 04:05:20 on dist-test-slave-1l3l"
server_key: "1ce4bdf903f75c6c895095095dc4d0e5"
server_key_iv: "1bdc7d30843e156a376fc71c6423e3ac"
server_key_version: "encryptionkey@0"
I20260808 04:05:20.037853 19969 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 04:05:20.041016 19992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:05:20.042232 19969 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260808 04:05:20.042356 19969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-2/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-2/wal
uuid: "724c8b24bc224ae58e2bb9d1f6d26c0e"
format_stamp: "Formatted at 2026-08-08 04:05:20 on dist-test-slave-1l3l"
server_key: "1ce4bdf903f75c6c895095095dc4d0e5"
server_key_iv: "1bdc7d30843e156a376fc71c6423e3ac"
server_key_version: "encryptionkey@0"
I20260808 04:05:20.042519 19969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:05:20.047763 19969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:05:20.048449 19969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:05:20.048686 19969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:05:20.049542 19969 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:05:20.050846 19969 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:05:20.050937 19969 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:05:20.051016 19969 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:05:20.051069 19969 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:05:20.080762 19969 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.3:32933
I20260808 04:05:20.080829 20154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.3:32933 every 8 connection(s)
I20260808 04:05:20.082018 19969 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
I20260808 04:05:20.089773 19761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:20.084588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.3:32951 (local address 127.5.83.62:38033)
0808 04:05:20.084806 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0808 04:05:20.084812 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:05:20.086114 (+ 1302us) server_negotiation.cc:407] Connection header received
0808 04:05:20.087743 (+ 1629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:05:20.087747 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:05:20.087798 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:05:20.087890 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:05:20.089191 (+ 1301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:05:20.089195 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:05:20.089198 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:05:20.089212 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:05:20.089242 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:05:20.089253 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:05:20.089350 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0808 04:05:20.089514 (+ 164us) server_negotiation.cc:299] Negotiation successful
0808 04:05:20.089560 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260808 04:05:20.090611 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 19969
I20260808 04:05:20.090672 20159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:20.085077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:38033 (local address 127.5.83.3:32951)
0808 04:05:20.085967 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0808 04:05:20.086003 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0808 04:05:20.087486 (+ 1483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:05:20.088092 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:05:20.088100 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:05:20.088184 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:05:20.088859 (+ 675us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:05:20.088862 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:05:20.088913 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:05:20.089059 (+ 146us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:05:20.089355 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:05:20.089362 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:05:20.089367 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 04:05:20.089573 (+ 206us) client_negotiation.cc:240] Negotiation successful
0808 04:05:20.089751 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":285,"threads_started":1}
I20260808 04:05:20.090780 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161826605655-5452-0/minicluster-data/ts-2/wal/instance
I20260808 04:05:20.091181 5452 external_mini_cluster.cc:1428] Setting key 36ce97d329dd7646a37abf2377eefacf
I20260808 04:05:20.092023 20155 heartbeater.cc:347] Connected to a master server at 127.5.83.62:38033
I20260808 04:05:20.092290 20155 heartbeater.cc:464] Registering TS with master...
I20260808 04:05:20.092844 20155 heartbeater.cc:511] Master 127.5.83.62:38033 requested a full tablet report, sending...
I20260808 04:05:20.094146 19458 ts_manager.cc:194] Registered new tserver with Master: 724c8b24bc224ae58e2bb9d1f6d26c0e (127.5.83.3:32933)
I20260808 04:05:20.105790 5452 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260808 04:05:20.111353 19761 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:05:20.110047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33118 (local address 127.5.83.62:38033)
0808 04:05:20.110186 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0808 04:05:20.110190 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:05:20.110668 (+ 478us) server_negotiation.cc:407] Connection header received
0808 04:05:20.110789 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:05:20.110793 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:05:20.110846 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:05:20.110923 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:05:20.111172 (+ 249us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33118: BlockingRecv error: recv got EOF from 127.0.0.1:33118 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 04:05:20.112270 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 19548
I20260808 04:05:20.115275 20167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:20.113108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44460 (local address 127.5.83.1:44783)
0808 04:05:20.113563 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0808 04:05:20.113569 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:05:20.113592 (+ 23us) server_negotiation.cc:407] Connection header received
0808 04:05:20.113644 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:05:20.113649 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:05:20.113812 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:05:20.113933 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:05:20.114760 (+ 827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:05:20.114766 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:05:20.114776 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:05:20.114788 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:05:20.114827 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:05:20.114844 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:05:20.114948 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0808 04:05:20.114992 (+ 44us) server_negotiation.cc:299] Negotiation successful
0808 04:05:20.115068 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":213,"threads_started":1}
I20260808 04:05:20.116370 19753 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:05:20.195429 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 19548
I20260808 04:05:20.217183 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 19766
I20260808 04:05:20.220441 20169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:20.218033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35256 (local address 127.5.83.2:45093)
0808 04:05:20.218554 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0808 04:05:20.218560 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:05:20.218588 (+ 28us) server_negotiation.cc:407] Connection header received
0808 04:05:20.218651 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:05:20.218657 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:05:20.218862 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:05:20.218990 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:05:20.219812 (+ 822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:05:20.219819 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:05:20.219831 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:05:20.219845 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:05:20.219895 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:05:20.219914 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:05:20.220020 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0808 04:05:20.220088 (+ 68us) server_negotiation.cc:299] Negotiation successful
0808 04:05:20.220173 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":219,"threads_started":1}
I20260808 04:05:20.221652 19955 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:05:20.300387 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 19766
I20260808 04:05:20.320559 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 19969
I20260808 04:05:20.324545 20171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:20.321502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50550 (local address 127.5.83.3:32933)
0808 04:05:20.322161 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0808 04:05:20.322168 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:05:20.322199 (+ 31us) server_negotiation.cc:407] Connection header received
0808 04:05:20.322262 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:05:20.322268 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:05:20.322487 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:05:20.322633 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:05:20.323792 (+ 1159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:05:20.323800 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:05:20.323810 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:05:20.323831 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:05:20.323871 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:05:20.323885 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:05:20.324052 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0808 04:05:20.324143 (+ 91us) server_negotiation.cc:299] Negotiation successful
0808 04:05:20.324247 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":326,"threads_started":1}
I20260808 04:05:20.325783 20149 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:05:20.404642 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 19969
I20260808 04:05:20.424377 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 19416
I20260808 04:05:20.425565 19501 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:05:20.525404 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 19416
2026-08-08T04:05:20Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2278 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:05:20 dist-test-slave-1l3l krb5kdc[20179](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:05:20 dist-test-slave-1l3l krb5kdc[20179](info): set up 2 sockets
Aug 08 04:05:20 dist-test-slave-1l3l krb5kdc[20179](info): commencing operation
krb5kdc: starting...
W20260808 04:05:21.721567 5452 mini_kdc.cc:121] Time spent starting KDC: real 1.152s user 0.002s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/krb5kdc/test-user.keytab.
Aug 08 04:05:21 dist-test-slave-1l3l krb5kdc[20179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:05:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:05:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.83.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab.
Entry for principal kudu/127.5.83.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab.
I20260808 04:05:21.951192 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:35223
--webserver_interface=127.5.83.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab
--principal=kudu/127.5.83.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:41085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.62:35223
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:05:22.259327 20199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:05:22.259588 20199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:05:22.259732 20199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:05:22.266570 20199 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:05:22.266667 20199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:05:22.266739 20199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:05:22.266809 20199 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:05:22.266871 20199 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:05:22.276057 20199 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:41085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.62:35223
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab
--principal=kudu/127.5.83.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:35223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:05:22.277233 20199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:05:22.278481 20199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:05:22.289251 20212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:05:22.289347 20210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:05:22.289327 20215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:05:22.290196 20199 server_base.cc:1034] running on GCE node
I20260808 04:05:22.290880 20199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:05:22.292166 20199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:05:22.293365 20199 hybrid_clock.cc:648] HybridClock initialized: now 1786161922293316 us; error 43 us; skew 500 ppm
I20260808 04:05:22.293666 20199 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:05:22 dist-test-slave-1l3l krb5kdc[20179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161922, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:05:22.300613 20199 init.cc:377] Logged in from keytab as kudu/127.5.83.62@KRBTEST.COM (short username kudu)
I20260808 04:05:22.301963 20199 webserver.cc:460] Webserver started at http://127.5.83.62:34347/ using document root <none> and password file <none>
I20260808 04:05:22.302590 20199 fs_manager.cc:359] Metadata directory not provided
I20260808 04:05:22.302707 20199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:05:22.302981 20199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:05:22.305786 20199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/minicluster-data/master-0/data/instance:
uuid: "6e60926fd0f14f59a77f63e7f8ffcccb"
format_stamp: "Formatted at 2026-08-08 04:05:22 on dist-test-slave-1l3l"
server_key: "ae33c112ee3cce7c35b8afd1ce56d600"
server_key_iv: "106b986528d898637e7f916f40d557fb"
server_key_version: "encryptionkey@0"
I20260808 04:05:22.306504 20199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/minicluster-data/master-0/wal/instance:
uuid: "6e60926fd0f14f59a77f63e7f8ffcccb"
format_stamp: "Formatted at 2026-08-08 04:05:22 on dist-test-slave-1l3l"
server_key: "ae33c112ee3cce7c35b8afd1ce56d600"
server_key_iv: "106b986528d898637e7f916f40d557fb"
server_key_version: "encryptionkey@0"
I20260808 04:05:22.311213 20199 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260808 04:05:22.314381 20222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:05:22.315603 20199 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260808 04:05:22.315770 20199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/minicluster-data/master-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/minicluster-data/master-0/wal
uuid: "6e60926fd0f14f59a77f63e7f8ffcccb"
format_stamp: "Formatted at 2026-08-08 04:05:22 on dist-test-slave-1l3l"
server_key: "ae33c112ee3cce7c35b8afd1ce56d600"
server_key_iv: "106b986528d898637e7f916f40d557fb"
server_key_version: "encryptionkey@0"
I20260808 04:05:22.315935 20199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:05:22.339679 20199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:05:22.343538 20199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:05:22.343832 20199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:05:22.360838 20199 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.62:35223
I20260808 04:05:22.360864 20314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.62:35223 every 8 connection(s)
I20260808 04:05:22.362110 20199 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
I20260808 04:05:22.365620 20315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:05:22.369657 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 20199
I20260808 04:05:22.369870 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/minicluster-data/master-0/wal/instance
I20260808 04:05:22.370474 5452 external_mini_cluster.cc:1428] Setting key 8419eb38c416e4561f9285fbe47cfc2a
I20260808 04:05:22.377245 20315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e60926fd0f14f59a77f63e7f8ffcccb: Bootstrap starting.
I20260808 04:05:22.380713 20315 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e60926fd0f14f59a77f63e7f8ffcccb: Neither blocks nor log segments found. Creating new log.
I20260808 04:05:22.381872 20315 log.cc:824] T 00000000000000000000000000000000 P 6e60926fd0f14f59a77f63e7f8ffcccb: Log is configured to *not* fsync() on all Append() calls
I20260808 04:05:22.384688 20315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e60926fd0f14f59a77f63e7f8ffcccb: No bootstrap required, opened a new log
Aug 08 04:05:22 dist-test-slave-1l3l krb5kdc[20179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:05:22.393831 20315 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e60926fd0f14f59a77f63e7f8ffcccb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e60926fd0f14f59a77f63e7f8ffcccb" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 35223 } }
I20260808 04:05:22.394173 20315 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e60926fd0f14f59a77f63e7f8ffcccb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:05:22.394274 20315 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e60926fd0f14f59a77f63e7f8ffcccb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e60926fd0f14f59a77f63e7f8ffcccb, State: Initialized, Role: FOLLOWER
I20260808 04:05:22.394447 20318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:22.372397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42884 (local address 127.5.83.62:35223)
0808 04:05:22.373182 (+ 785us) server_negotiation.cc:206] Beginning negotiation
0808 04:05:22.373201 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 04:05:22.373255 (+ 54us) server_negotiation.cc:407] Connection header received
0808 04:05:22.375647 (+ 2392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:05:22.375674 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:05:22.376157 (+ 483us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:05:22.376559 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:05:22.377915 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:22.379933 (+ 2018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:22.381194 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:22.382209 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:22.382407 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:22.386624 (+ 4217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:05:22.386635 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:05:22.386653 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:05:22.386697 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:05:22.390567 (+ 3870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:05:22.391227 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:05:22.391235 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:05:22.391243 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:05:22.391448 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:05:22.391797 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:05:22.391802 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:05:22.391806 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:05:22.392343 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:05:22.392658 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0808 04:05:22.393075 (+ 417us) server_negotiation.cc:299] Negotiation successful
0808 04:05:22.393449 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":302,"threads_started":1}
I20260808 04:05:22.394850 20315 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e60926fd0f14f59a77f63e7f8ffcccb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e60926fd0f14f59a77f63e7f8ffcccb" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 35223 } }
I20260808 04:05:22.395089 20315 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e60926fd0f14f59a77f63e7f8ffcccb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:05:22.395213 20315 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e60926fd0f14f59a77f63e7f8ffcccb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:05:22.395386 20315 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e60926fd0f14f59a77f63e7f8ffcccb [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:05:22.397789 20315 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e60926fd0f14f59a77f63e7f8ffcccb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e60926fd0f14f59a77f63e7f8ffcccb" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 35223 } }
I20260808 04:05:22.398221 20315 leader_election.cc:302] T 00000000000000000000000000000000 P 6e60926fd0f14f59a77f63e7f8ffcccb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e60926fd0f14f59a77f63e7f8ffcccb; no voters:
I20260808 04:05:22.398753 20315 leader_election.cc:288] T 00000000000000000000000000000000 P 6e60926fd0f14f59a77f63e7f8ffcccb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:05:22.398857 20320 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e60926fd0f14f59a77f63e7f8ffcccb [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:05:22.399458 20320 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e60926fd0f14f59a77f63e7f8ffcccb [term 1 LEADER]: Becoming Leader. State: Replica: 6e60926fd0f14f59a77f63e7f8ffcccb, State: Running, Role: LEADER
I20260808 04:05:22.399973 20320 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e60926fd0f14f59a77f63e7f8ffcccb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e60926fd0f14f59a77f63e7f8ffcccb" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 35223 } }
I20260808 04:05:22.400607 20315 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e60926fd0f14f59a77f63e7f8ffcccb [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:05:22.402603 20321 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e60926fd0f14f59a77f63e7f8ffcccb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e60926fd0f14f59a77f63e7f8ffcccb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e60926fd0f14f59a77f63e7f8ffcccb" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 35223 } } }
I20260808 04:05:22.403056 20321 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e60926fd0f14f59a77f63e7f8ffcccb [sys.catalog]: This master's current role is: LEADER
I20260808 04:05:22.402957 20322 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e60926fd0f14f59a77f63e7f8ffcccb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e60926fd0f14f59a77f63e7f8ffcccb. Latest consensus state: current_term: 1 leader_uuid: "6e60926fd0f14f59a77f63e7f8ffcccb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e60926fd0f14f59a77f63e7f8ffcccb" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 35223 } } }
I20260808 04:05:22.403288 20322 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e60926fd0f14f59a77f63e7f8ffcccb [sys.catalog]: This master's current role is: LEADER
I20260808 04:05:22.405066 20331 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:05:22.409132 20331 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:05:22.417001 20331 catalog_manager.cc:1347] Generated new cluster ID: f5a7036262574e44b81c57127578f223
I20260808 04:05:22.417090 20331 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:05:22.434556 20331 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:05:22.435801 20331 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:05:22.447762 20331 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e60926fd0f14f59a77f63e7f8ffcccb: Generated new TSK 0
I20260808 04:05:22.448518 20331 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.83.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.1.keytab.
Entry for principal kudu/127.5.83.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.1.keytab.
I20260808 04:05:22.503168 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.1:0
--local_ip_for_outbound_sockets=127.5.83.1
--webserver_interface=127.5.83.1
--webserver_port=0
--tserver_master_addrs=127.5.83.62:35223
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.1.keytab
--principal=kudu/127.5.83.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:41085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:05:22.821695 20349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:05:22.821959 20349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:05:22.822088 20349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:05:22.829196 20349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:05:22.829303 20349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:05:22.829478 20349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.1
I20260808 04:05:22.838845 20349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:41085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.1.keytab
--principal=kudu/127.5.83.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.83.1
--webserver_port=0
--tserver_master_addrs=127.5.83.62:35223
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.1
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:05:22.840065 20349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:05:22.841323 20349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:05:22.853488 20364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:05:22.853615 20358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:05:22.853549 20360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:05:22.854533 20349 server_base.cc:1034] running on GCE node
I20260808 04:05:22.855072 20349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:05:22.855942 20349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:05:22.857136 20349 hybrid_clock.cc:648] HybridClock initialized: now 1786161922857090 us; error 40 us; skew 500 ppm
I20260808 04:05:22.857429 20349 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:05:22 dist-test-slave-1l3l krb5kdc[20179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161922, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:05:22.860415 20349 init.cc:377] Logged in from keytab as kudu/127.5.83.1@KRBTEST.COM (short username kudu)
I20260808 04:05:22.861694 20349 webserver.cc:460] Webserver started at http://127.5.83.1:46805/ using document root <none> and password file <none>
I20260808 04:05:22.862305 20349 fs_manager.cc:359] Metadata directory not provided
I20260808 04:05:22.862429 20349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:05:22.862695 20349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:05:22.865485 20349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/minicluster-data/ts-0/data/instance:
uuid: "9b42bba36bc44ed6a2f77dfe1716befb"
format_stamp: "Formatted at 2026-08-08 04:05:22 on dist-test-slave-1l3l"
server_key: "9ea957e42767ad911df6d387eed16dd5"
server_key_iv: "e267b721b9cedf146c5012ac4b572b33"
server_key_version: "encryptionkey@0"
I20260808 04:05:22.866227 20349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/minicluster-data/ts-0/wal/instance:
uuid: "9b42bba36bc44ed6a2f77dfe1716befb"
format_stamp: "Formatted at 2026-08-08 04:05:22 on dist-test-slave-1l3l"
server_key: "9ea957e42767ad911df6d387eed16dd5"
server_key_iv: "e267b721b9cedf146c5012ac4b572b33"
server_key_version: "encryptionkey@0"
I20260808 04:05:22.871008 20349 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 04:05:22.874204 20374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:05:22.875389 20349 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:05:22.875537 20349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/minicluster-data/ts-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/minicluster-data/ts-0/wal
uuid: "9b42bba36bc44ed6a2f77dfe1716befb"
format_stamp: "Formatted at 2026-08-08 04:05:22 on dist-test-slave-1l3l"
server_key: "9ea957e42767ad911df6d387eed16dd5"
server_key_iv: "e267b721b9cedf146c5012ac4b572b33"
server_key_version: "encryptionkey@0"
I20260808 04:05:22.875767 20349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:05:22.904906 20349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:05:22.908986 20349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:05:22.909277 20349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:05:22.910308 20349 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:05:22.911609 20349 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:05:22.911687 20349 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:05:22.911826 20349 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:05:22.911887 20349 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:05:22.942356 20349 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.1:35651
I20260808 04:05:22.942394 20545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.1:35651 every 8 connection(s)
I20260808 04:05:22.943573 20349 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
I20260808 04:05:22.944835 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 20349
I20260808 04:05:22.945057 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/minicluster-data/ts-0/wal/instance
I20260808 04:05:22.945662 5452 external_mini_cluster.cc:1428] Setting key b4837dce0d4d87bb37dcf9adc4fb47ff
Aug 08 04:05:22 dist-test-slave-1l3l krb5kdc[20179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161922, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.1@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:05:22.962682 20548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:22.946216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.1:43205 (local address 127.5.83.62:35223)
0808 04:05:22.946684 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0808 04:05:22.946688 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:05:22.948444 (+ 1756us) server_negotiation.cc:407] Connection header received
0808 04:05:22.949596 (+ 1152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:05:22.949602 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:05:22.949689 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:05:22.949802 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:05:22.951230 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:22.951977 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:22.953476 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:22.953882 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:22.953998 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:22.957229 (+ 3231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:05:22.957235 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:05:22.957238 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:05:22.957268 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:05:22.959816 (+ 2548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:05:22.960590 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:05:22.960594 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:05:22.960596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:05:22.960715 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:05:22.961180 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:05:22.961183 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:05:22.961185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:05:22.961436 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:05:22.961541 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0808 04:05:22.962271 (+ 730us) server_negotiation.cc:299] Negotiation successful
0808 04:05:22.962441 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":241,"threads_started":1}
I20260808 04:05:22.963997 20549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:22.946774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:35223 (local address 127.5.83.1:43205)
0808 04:05:22.947559 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0808 04:05:22.947604 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 04:05:22.949342 (+ 1738us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:05:22.950030 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:05:22.950038 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:05:22.950521 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:05:22.951067 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:22.951084 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:22.952140 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:22.952144 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:22.953328 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:22.953341 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:22.954101 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:22.954109 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:22.954323 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:22.954936 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:05:22.954957 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:05:22.957029 (+ 2072us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:05:22.959948 (+ 2919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:05:22.959957 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:05:22.959971 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:05:22.960495 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:05:22.960819 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:05:22.960823 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:05:22.960826 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:05:22.961085 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:05:22.961552 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:05:22.961559 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:05:22.961922 (+ 363us) client_negotiation.cc:769] Sending connection context
0808 04:05:22.962189 (+ 267us) client_negotiation.cc:240] Negotiation successful
0808 04:05:22.962734 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":321,"threads_started":1}
I20260808 04:05:22.965442 20546 heartbeater.cc:347] Connected to a master server at 127.5.83.62:35223
I20260808 04:05:22.965736 20546 heartbeater.cc:464] Registering TS with master...
I20260808 04:05:22.966571 20546 heartbeater.cc:511] Master 127.5.83.62:35223 requested a full tablet report, sending...
I20260808 04:05:22.968601 20247 ts_manager.cc:194] Registered new tserver with Master: 9b42bba36bc44ed6a2f77dfe1716befb (127.5.83.1:35651)
I20260808 04:05:22.970307 20247 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.1@KRBTEST.COM'} at 127.5.83.1:43205
WARNING: no policy specified for kudu/127.5.83.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.2.keytab.
Entry for principal kudu/127.5.83.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.2.keytab.
I20260808 04:05:23.004585 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.2:0
--local_ip_for_outbound_sockets=127.5.83.2
--webserver_interface=127.5.83.2
--webserver_port=0
--tserver_master_addrs=127.5.83.62:35223
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.2.keytab
--principal=kudu/127.5.83.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:41085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:05:23.324447 20552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:05:23.324704 20552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:05:23.324797 20552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:05:23.331730 20552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:05:23.331823 20552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:05:23.331938 20552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.2
I20260808 04:05:23.341069 20552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:41085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.2.keytab
--principal=kudu/127.5.83.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.83.2
--webserver_port=0
--tserver_master_addrs=127.5.83.62:35223
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.2
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:05:23.342141 20552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:05:23.343333 20552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:05:23.355474 20558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:05:23.355433 20561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:05:23.355557 20559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:05:23.356530 20552 server_base.cc:1034] running on GCE node
I20260808 04:05:23.357041 20552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:05:23.357859 20552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:05:23.359050 20552 hybrid_clock.cc:648] HybridClock initialized: now 1786161923359004 us; error 37 us; skew 500 ppm
I20260808 04:05:23.359338 20552 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:05:23 dist-test-slave-1l3l krb5kdc[20179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161923, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:05:23.362358 20552 init.cc:377] Logged in from keytab as kudu/127.5.83.2@KRBTEST.COM (short username kudu)
I20260808 04:05:23.363756 20552 webserver.cc:460] Webserver started at http://127.5.83.2:41223/ using document root <none> and password file <none>
I20260808 04:05:23.364362 20552 fs_manager.cc:359] Metadata directory not provided
I20260808 04:05:23.364466 20552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:05:23.364733 20552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:05:23.367430 20552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/minicluster-data/ts-1/data/instance:
uuid: "aae4238078a346ffb86fddbf8b4da39f"
format_stamp: "Formatted at 2026-08-08 04:05:23 on dist-test-slave-1l3l"
server_key: "e0dc6209646033b79797f4ff486ca282"
server_key_iv: "9bc75ce3c076399689a6b5fefbbd6ec2"
server_key_version: "encryptionkey@0"
I20260808 04:05:23.368270 20552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/minicluster-data/ts-1/wal/instance:
uuid: "aae4238078a346ffb86fddbf8b4da39f"
format_stamp: "Formatted at 2026-08-08 04:05:23 on dist-test-slave-1l3l"
server_key: "e0dc6209646033b79797f4ff486ca282"
server_key_iv: "9bc75ce3c076399689a6b5fefbbd6ec2"
server_key_version: "encryptionkey@0"
I20260808 04:05:23.373044 20552 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260808 04:05:23.376349 20574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:05:23.377532 20552 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.002s
I20260808 04:05:23.377671 20552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/minicluster-data/ts-1/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/minicluster-data/ts-1/wal
uuid: "aae4238078a346ffb86fddbf8b4da39f"
format_stamp: "Formatted at 2026-08-08 04:05:23 on dist-test-slave-1l3l"
server_key: "e0dc6209646033b79797f4ff486ca282"
server_key_iv: "9bc75ce3c076399689a6b5fefbbd6ec2"
server_key_version: "encryptionkey@0"
I20260808 04:05:23.377837 20552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:05:23.399569 20552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:05:23.403430 20552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:05:23.403671 20552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:05:23.404567 20552 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:05:23.405787 20552 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:05:23.405866 20552 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:05:23.405977 20552 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:05:23.406041 20552 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:05:23.435958 20552 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.2:40451
I20260808 04:05:23.436018 20744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.2:40451 every 8 connection(s)
I20260808 04:05:23.437177 20552 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
I20260808 04:05:23.446883 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 20552
I20260808 04:05:23.447067 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/minicluster-data/ts-1/wal/instance
I20260808 04:05:23.447630 5452 external_mini_cluster.cc:1428] Setting key caf648234e4a199dbdbdded5624688a8
Aug 08 04:05:23 dist-test-slave-1l3l krb5kdc[20179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161923, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.2@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:05:23.456787 20548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:23.439619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.2:48531 (local address 127.5.83.62:35223)
0808 04:05:23.439856 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0808 04:05:23.439861 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:05:23.440904 (+ 1043us) server_negotiation.cc:407] Connection header received
0808 04:05:23.442617 (+ 1713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:05:23.442621 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:05:23.442706 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:05:23.442793 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:05:23.444312 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:23.445053 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:23.446556 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:23.446949 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:23.447065 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:23.451097 (+ 4032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:05:23.451102 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:05:23.451105 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:05:23.451136 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:05:23.453701 (+ 2565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:05:23.454529 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:05:23.454533 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:05:23.454536 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:05:23.454666 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:05:23.455114 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:05:23.455118 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:05:23.455120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:05:23.455411 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:05:23.455522 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0808 04:05:23.456381 (+ 859us) server_negotiation.cc:299] Negotiation successful
0808 04:05:23.456551 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260808 04:05:23.457981 20747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:23.440033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:35223 (local address 127.5.83.2:48531)
0808 04:05:23.440739 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0808 04:05:23.440778 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 04:05:23.442369 (+ 1591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:05:23.443001 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:05:23.443009 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:05:23.443527 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:05:23.444167 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:23.444190 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:23.445203 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:23.445207 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:23.446425 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:23.446437 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:23.447173 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:23.447181 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:23.447476 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:23.448243 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:05:23.448265 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:05:23.450857 (+ 2592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:05:23.453841 (+ 2984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:05:23.453849 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:05:23.453862 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:05:23.454380 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:05:23.454765 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:05:23.454769 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:05:23.454772 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:05:23.455006 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:05:23.455530 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:05:23.455537 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:05:23.455956 (+ 419us) client_negotiation.cc:769] Sending connection context
0808 04:05:23.456231 (+ 275us) client_negotiation.cc:240] Negotiation successful
0808 04:05:23.456710 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":278,"threads_started":1}
I20260808 04:05:23.459441 20745 heartbeater.cc:347] Connected to a master server at 127.5.83.62:35223
I20260808 04:05:23.459748 20745 heartbeater.cc:464] Registering TS with master...
I20260808 04:05:23.460461 20745 heartbeater.cc:511] Master 127.5.83.62:35223 requested a full tablet report, sending...
I20260808 04:05:23.461975 20247 ts_manager.cc:194] Registered new tserver with Master: aae4238078a346ffb86fddbf8b4da39f (127.5.83.2:40451)
I20260808 04:05:23.462766 20247 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.2@KRBTEST.COM'} at 127.5.83.2:48531
WARNING: no policy specified for kudu/127.5.83.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.3.keytab.
Entry for principal kudu/127.5.83.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.3.keytab.
I20260808 04:05:23.502106 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.3:0
--local_ip_for_outbound_sockets=127.5.83.3
--webserver_interface=127.5.83.3
--webserver_port=0
--tserver_master_addrs=127.5.83.62:35223
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.3.keytab
--principal=kudu/127.5.83.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:41085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:05:23.816272 20752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:05:23.816527 20752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:05:23.816650 20752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:05:23.823629 20752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:05:23.823743 20752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:05:23.823904 20752 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.3
I20260808 04:05:23.833205 20752 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:41085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.3.keytab
--principal=kudu/127.5.83.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.83.3
--webserver_port=0
--tserver_master_addrs=127.5.83.62:35223
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.3
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:05:23.834396 20752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:05:23.835662 20752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:05:23.847856 20764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:05:23.847951 20766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:05:23.848353 20763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:05:23.848893 20752 server_base.cc:1034] running on GCE node
I20260808 04:05:23.849416 20752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:05:23.850245 20752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:05:23.851437 20752 hybrid_clock.cc:648] HybridClock initialized: now 1786161923851385 us; error 45 us; skew 500 ppm
I20260808 04:05:23.851732 20752 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:05:23 dist-test-slave-1l3l krb5kdc[20179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161923, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:05:23.854650 20752 init.cc:377] Logged in from keytab as kudu/127.5.83.3@KRBTEST.COM (short username kudu)
I20260808 04:05:23.855986 20752 webserver.cc:460] Webserver started at http://127.5.83.3:39357/ using document root <none> and password file <none>
I20260808 04:05:23.856590 20752 fs_manager.cc:359] Metadata directory not provided
I20260808 04:05:23.856706 20752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:05:23.856973 20752 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:05:23.859658 20752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/minicluster-data/ts-2/data/instance:
uuid: "6efc6cb40fe94d259fdddadf6d4f7bc8"
format_stamp: "Formatted at 2026-08-08 04:05:23 on dist-test-slave-1l3l"
server_key: "5aa859d3398e96278721e41a67420186"
server_key_iv: "488210f01ed0c0dd2fb4e6e15f367d52"
server_key_version: "encryptionkey@0"
I20260808 04:05:23.860414 20752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/minicluster-data/ts-2/wal/instance:
uuid: "6efc6cb40fe94d259fdddadf6d4f7bc8"
format_stamp: "Formatted at 2026-08-08 04:05:23 on dist-test-slave-1l3l"
server_key: "5aa859d3398e96278721e41a67420186"
server_key_iv: "488210f01ed0c0dd2fb4e6e15f367d52"
server_key_version: "encryptionkey@0"
I20260808 04:05:23.865159 20752 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260808 04:05:23.868366 20773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:05:23.869576 20752 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:05:23.869722 20752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/minicluster-data/ts-2/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/minicluster-data/ts-2/wal
uuid: "6efc6cb40fe94d259fdddadf6d4f7bc8"
format_stamp: "Formatted at 2026-08-08 04:05:23 on dist-test-slave-1l3l"
server_key: "5aa859d3398e96278721e41a67420186"
server_key_iv: "488210f01ed0c0dd2fb4e6e15f367d52"
server_key_version: "encryptionkey@0"
I20260808 04:05:23.869895 20752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:05:23.888106 20752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:05:23.892009 20752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:05:23.892256 20752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:05:23.893131 20752 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:05:23.894335 20752 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:05:23.894423 20752 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:05:23.894542 20752 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:05:23.894598 20752 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:05:23.922148 20752 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.3:46599
I20260808 04:05:23.922183 20953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.3:46599 every 8 connection(s)
I20260808 04:05:23.923357 20752 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
I20260808 04:05:23.932453 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 20752
I20260808 04:05:23.932633 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/minicluster-data/ts-2/wal/instance
I20260808 04:05:23.933037 5452 external_mini_cluster.cc:1428] Setting key 708273f913a4bc0dad0bce304d682bac
Aug 08 04:05:23 dist-test-slave-1l3l krb5kdc[20179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161923, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.3@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:05:23.941097 20548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:23.925704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.3:56645 (local address 127.5.83.62:35223)
0808 04:05:23.925904 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0808 04:05:23.925911 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:05:23.926992 (+ 1081us) server_negotiation.cc:407] Connection header received
0808 04:05:23.928665 (+ 1673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:05:23.928670 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:05:23.928737 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:05:23.928821 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:05:23.930116 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:23.930856 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:23.932540 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:23.932980 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:23.933098 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:23.936490 (+ 3392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:05:23.936495 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:05:23.936499 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:05:23.936529 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:05:23.938710 (+ 2181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:05:23.939315 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:05:23.939319 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:05:23.939321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:05:23.939443 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:05:23.939789 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:05:23.939792 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:05:23.939794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:05:23.940044 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:05:23.940152 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0808 04:05:23.940690 (+ 538us) server_negotiation.cc:299] Negotiation successful
0808 04:05:23.940844 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 04:05:23.941962 20957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:23.926175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:35223 (local address 127.5.83.3:56645)
0808 04:05:23.926842 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0808 04:05:23.926879 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0808 04:05:23.928438 (+ 1559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:05:23.928999 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:05:23.929006 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:05:23.929478 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:05:23.929990 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:23.930006 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:23.930974 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:23.930978 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:23.932390 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:23.932404 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:23.933196 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:23.933203 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:23.933429 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:23.934021 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:05:23.934048 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:05:23.936313 (+ 2265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:05:23.938816 (+ 2503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:05:23.938822 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:05:23.938832 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:05:23.939227 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:05:23.939518 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:05:23.939521 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:05:23.939523 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:05:23.939681 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:05:23.940136 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:05:23.940141 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:05:23.940405 (+ 264us) client_negotiation.cc:769] Sending connection context
0808 04:05:23.940608 (+ 203us) client_negotiation.cc:240] Negotiation successful
0808 04:05:23.940946 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":265,"threads_started":1}
I20260808 04:05:23.943140 20954 heartbeater.cc:347] Connected to a master server at 127.5.83.62:35223
I20260808 04:05:23.943420 20954 heartbeater.cc:464] Registering TS with master...
I20260808 04:05:23.944257 20954 heartbeater.cc:511] Master 127.5.83.62:35223 requested a full tablet report, sending...
I20260808 04:05:23.945729 20247 ts_manager.cc:194] Registered new tserver with Master: 6efc6cb40fe94d259fdddadf6d4f7bc8 (127.5.83.3:46599)
I20260808 04:05:23.946642 20247 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.3@KRBTEST.COM'} at 127.5.83.3:56645
I20260808 04:05:23.948334 5452 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:05:23.964553 20548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:23.952678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42892 (local address 127.5.83.62:35223)
0808 04:05:23.952833 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0808 04:05:23.952839 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:05:23.953502 (+ 663us) server_negotiation.cc:407] Connection header received
0808 04:05:23.953656 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:05:23.953661 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:05:23.953717 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:05:23.953794 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:05:23.955110 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:23.955845 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:23.957031 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:23.957404 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:23.957510 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:23.959788 (+ 2278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:05:23.959795 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:05:23.959798 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:05:23.959827 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:05:23.962122 (+ 2295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:05:23.962723 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:05:23.962728 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:05:23.962732 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:05:23.962847 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:05:23.963188 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:05:23.963193 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:05:23.963197 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:05:23.963477 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:05:23.963696 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0808 04:05:23.964066 (+ 370us) server_negotiation.cc:299] Negotiation successful
0808 04:05:23.964274 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 04:05:23.967957 20247 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42892:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260808 04:05:23.973500 20546 heartbeater.cc:503] Master 127.5.83.62:35223 was elected leader, sending a full tablet report...
W20260808 04:05:23.974649 20247 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:05:23.996647 20969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:23.985433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.3:46599 (local address 127.0.0.1:36540)
0808 04:05:23.986668 (+ 1235us) negotiation.cc:107] Waiting for socket to connect
0808 04:05:23.986703 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0808 04:05:23.986859 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:05:23.988042 (+ 1183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:05:23.988053 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:05:23.988094 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:05:23.988531 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:23.988546 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:23.990779 (+ 2233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:23.990785 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:23.993283 (+ 2498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:23.993295 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:23.995506 (+ 2211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:23.995513 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:23.995768 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:23.995822 (+ 54us) client_negotiation.cc:769] Sending connection context
0808 04:05:23.995959 (+ 137us) client_negotiation.cc:240] Negotiation successful
0808 04:05:23.996151 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1015,"mutex_wait_us":1,"thread_start_us":2541,"threads_started":1}
I20260808 04:05:23.997448 20968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:23.985724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36540 (local address 127.5.83.3:46599)
0808 04:05:23.987522 (+ 1798us) server_negotiation.cc:206] Beginning negotiation
0808 04:05:23.987528 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:05:23.987554 (+ 26us) server_negotiation.cc:407] Connection header received
0808 04:05:23.987624 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:05:23.987629 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:05:23.987849 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:05:23.988010 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:05:23.988761 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:23.990610 (+ 1849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:23.993488 (+ 2878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:23.994972 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:23.995100 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:23.995353 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0808 04:05:23.996889 (+ 1536us) server_negotiation.cc:299] Negotiation successful
0808 04:05:23.997099 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1660,"thread_start_us":1524,"threads_started":1}
I20260808 04:05:23.998039 20971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:23.986574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40760 (local address 127.5.83.2:40451)
0808 04:05:23.987829 (+ 1255us) server_negotiation.cc:206] Beginning negotiation
0808 04:05:23.987837 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:05:23.988372 (+ 535us) server_negotiation.cc:407] Connection header received
0808 04:05:23.988685 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:05:23.988693 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:05:23.988927 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:05:23.989083 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:05:23.990087 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:23.991669 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:23.994341 (+ 2672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:23.995861 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:23.995972 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:23.996156 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0808 04:05:23.997497 (+ 1341us) server_negotiation.cc:299] Negotiation successful
0808 04:05:23.997689 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1080,"thread_start_us":963,"threads_started":1}
I20260808 04:05:23.997361 20967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:23.986081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.2:40451 (local address 127.0.0.1:40760)
0808 04:05:23.988268 (+ 2187us) negotiation.cc:107] Waiting for socket to connect
0808 04:05:23.988285 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:05:23.988482 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:05:23.989372 (+ 890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:05:23.989377 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:05:23.989402 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:05:23.989894 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:23.989904 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:23.991878 (+ 1974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:23.991893 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:23.994150 (+ 2257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:23.994167 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:23.996374 (+ 2207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:23.996380 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:23.996575 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:23.996612 (+ 37us) client_negotiation.cc:769] Sending connection context
0808 04:05:23.996715 (+ 103us) client_negotiation.cc:240] Negotiation successful
0808 04:05:23.996854 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2043,"mutex_wait_us":28,"thread_start_us":305,"threads_started":1}
I20260808 04:05:24.002377 20642 tablet_service.cc:1467] Processing CreateTablet for tablet 8b5ab39a877f42569cfed68d4a9c324b (DEFAULT_TABLE table=test-table [id=52a97a249f9141ff87bd60f7f33c95b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:05:24.002653 20841 tablet_service.cc:1467] Processing CreateTablet for tablet 8b5ab39a877f42569cfed68d4a9c324b (DEFAULT_TABLE table=test-table [id=52a97a249f9141ff87bd60f7f33c95b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:05:24.003952 20642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b5ab39a877f42569cfed68d4a9c324b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:05:24.003952 20841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b5ab39a877f42569cfed68d4a9c324b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:05:24.004858 20969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:23.988338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:35651 (local address 127.0.0.1:59544)
0808 04:05:23.997721 (+ 9383us) negotiation.cc:107] Waiting for socket to connect
0808 04:05:23.997737 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:05:23.997825 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:05:23.999768 (+ 1943us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:05:23.999773 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:05:23.999798 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:05:23.999952 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:23.999962 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:24.001253 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:24.001300 (+ 47us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:24.002908 (+ 1608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:24.002920 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:24.004158 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:24.004163 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:24.004334 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:24.004366 (+ 32us) client_negotiation.cc:769] Sending connection context
0808 04:05:24.004455 (+ 89us) client_negotiation.cc:240] Negotiation successful
0808 04:05:24.004595 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9308,"thread_start_us":10837,"threads_started":1}
I20260808 04:05:24.004961 20973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:23.988568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59544 (local address 127.5.83.1:35651)
0808 04:05:23.999206 (+ 10638us) server_negotiation.cc:206] Beginning negotiation
0808 04:05:23.999213 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:05:23.999238 (+ 25us) server_negotiation.cc:407] Connection header received
0808 04:05:23.999325 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:05:23.999333 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:05:23.999557 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:05:23.999762 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:05:24.000101 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:24.001118 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:24.003047 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:24.004012 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:24.004127 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:24.004342 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0808 04:05:24.004507 (+ 165us) server_negotiation.cc:299] Negotiation successful
0808 04:05:24.004655 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10461,"thread_start_us":10282,"threads_started":1}
I20260808 04:05:24.007613 20445 tablet_service.cc:1467] Processing CreateTablet for tablet 8b5ab39a877f42569cfed68d4a9c324b (DEFAULT_TABLE table=test-table [id=52a97a249f9141ff87bd60f7f33c95b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:05:24.008947 20445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b5ab39a877f42569cfed68d4a9c324b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:05:24.019325 20979 tablet_bootstrap.cc:492] T 8b5ab39a877f42569cfed68d4a9c324b P aae4238078a346ffb86fddbf8b4da39f: Bootstrap starting.
I20260808 04:05:24.019538 20978 tablet_bootstrap.cc:492] T 8b5ab39a877f42569cfed68d4a9c324b P 6efc6cb40fe94d259fdddadf6d4f7bc8: Bootstrap starting.
I20260808 04:05:24.024016 20979 tablet_bootstrap.cc:654] T 8b5ab39a877f42569cfed68d4a9c324b P aae4238078a346ffb86fddbf8b4da39f: Neither blocks nor log segments found. Creating new log.
I20260808 04:05:24.024932 20980 tablet_bootstrap.cc:492] T 8b5ab39a877f42569cfed68d4a9c324b P 9b42bba36bc44ed6a2f77dfe1716befb: Bootstrap starting.
I20260808 04:05:24.025352 20979 log.cc:824] T 8b5ab39a877f42569cfed68d4a9c324b P aae4238078a346ffb86fddbf8b4da39f: Log is configured to *not* fsync() on all Append() calls
I20260808 04:05:24.027737 20978 tablet_bootstrap.cc:654] T 8b5ab39a877f42569cfed68d4a9c324b P 6efc6cb40fe94d259fdddadf6d4f7bc8: Neither blocks nor log segments found. Creating new log.
I20260808 04:05:24.029035 20978 log.cc:824] T 8b5ab39a877f42569cfed68d4a9c324b P 6efc6cb40fe94d259fdddadf6d4f7bc8: Log is configured to *not* fsync() on all Append() calls
I20260808 04:05:24.029189 20980 tablet_bootstrap.cc:654] T 8b5ab39a877f42569cfed68d4a9c324b P 9b42bba36bc44ed6a2f77dfe1716befb: Neither blocks nor log segments found. Creating new log.
I20260808 04:05:24.030378 20980 log.cc:824] T 8b5ab39a877f42569cfed68d4a9c324b P 9b42bba36bc44ed6a2f77dfe1716befb: Log is configured to *not* fsync() on all Append() calls
I20260808 04:05:24.030803 20979 tablet_bootstrap.cc:492] T 8b5ab39a877f42569cfed68d4a9c324b P aae4238078a346ffb86fddbf8b4da39f: No bootstrap required, opened a new log
I20260808 04:05:24.031145 20979 ts_tablet_manager.cc:1397] T 8b5ab39a877f42569cfed68d4a9c324b P aae4238078a346ffb86fddbf8b4da39f: Time spent bootstrapping tablet: real 0.012s user 0.011s sys 0.000s
I20260808 04:05:24.031605 20978 tablet_bootstrap.cc:492] T 8b5ab39a877f42569cfed68d4a9c324b P 6efc6cb40fe94d259fdddadf6d4f7bc8: No bootstrap required, opened a new log
I20260808 04:05:24.031908 20978 ts_tablet_manager.cc:1397] T 8b5ab39a877f42569cfed68d4a9c324b P 6efc6cb40fe94d259fdddadf6d4f7bc8: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.005s
I20260808 04:05:24.033419 20980 tablet_bootstrap.cc:492] T 8b5ab39a877f42569cfed68d4a9c324b P 9b42bba36bc44ed6a2f77dfe1716befb: No bootstrap required, opened a new log
I20260808 04:05:24.033663 20980 ts_tablet_manager.cc:1397] T 8b5ab39a877f42569cfed68d4a9c324b P 9b42bba36bc44ed6a2f77dfe1716befb: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.005s
I20260808 04:05:24.039381 20978 raft_consensus.cc:348] T 8b5ab39a877f42569cfed68d4a9c324b P 6efc6cb40fe94d259fdddadf6d4f7bc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efc6cb40fe94d259fdddadf6d4f7bc8" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 46599 } } peers { permanent_uuid: "9b42bba36bc44ed6a2f77dfe1716befb" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 35651 } } peers { permanent_uuid: "aae4238078a346ffb86fddbf8b4da39f" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 40451 } }
I20260808 04:05:24.039763 20978 raft_consensus.cc:374] T 8b5ab39a877f42569cfed68d4a9c324b P 6efc6cb40fe94d259fdddadf6d4f7bc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:05:24.039865 20978 raft_consensus.cc:729] T 8b5ab39a877f42569cfed68d4a9c324b P 6efc6cb40fe94d259fdddadf6d4f7bc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6efc6cb40fe94d259fdddadf6d4f7bc8, State: Initialized, Role: FOLLOWER
I20260808 04:05:24.040412 20978 consensus_queue.cc:260] T 8b5ab39a877f42569cfed68d4a9c324b P 6efc6cb40fe94d259fdddadf6d4f7bc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efc6cb40fe94d259fdddadf6d4f7bc8" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 46599 } } peers { permanent_uuid: "9b42bba36bc44ed6a2f77dfe1716befb" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 35651 } } peers { permanent_uuid: "aae4238078a346ffb86fddbf8b4da39f" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 40451 } }
I20260808 04:05:24.041407 20954 heartbeater.cc:503] Master 127.5.83.62:35223 was elected leader, sending a full tablet report...
I20260808 04:05:24.041158 20979 raft_consensus.cc:348] T 8b5ab39a877f42569cfed68d4a9c324b P aae4238078a346ffb86fddbf8b4da39f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efc6cb40fe94d259fdddadf6d4f7bc8" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 46599 } } peers { permanent_uuid: "9b42bba36bc44ed6a2f77dfe1716befb" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 35651 } } peers { permanent_uuid: "aae4238078a346ffb86fddbf8b4da39f" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 40451 } }
I20260808 04:05:24.041566 20979 raft_consensus.cc:374] T 8b5ab39a877f42569cfed68d4a9c324b P aae4238078a346ffb86fddbf8b4da39f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:05:24.041675 20979 raft_consensus.cc:729] T 8b5ab39a877f42569cfed68d4a9c324b P aae4238078a346ffb86fddbf8b4da39f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae4238078a346ffb86fddbf8b4da39f, State: Initialized, Role: FOLLOWER
I20260808 04:05:24.041949 20978 ts_tablet_manager.cc:1428] T 8b5ab39a877f42569cfed68d4a9c324b P 6efc6cb40fe94d259fdddadf6d4f7bc8: Time spent starting tablet: real 0.010s user 0.006s sys 0.003s
I20260808 04:05:24.042228 20979 consensus_queue.cc:260] T 8b5ab39a877f42569cfed68d4a9c324b P aae4238078a346ffb86fddbf8b4da39f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efc6cb40fe94d259fdddadf6d4f7bc8" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 46599 } } peers { permanent_uuid: "9b42bba36bc44ed6a2f77dfe1716befb" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 35651 } } peers { permanent_uuid: "aae4238078a346ffb86fddbf8b4da39f" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 40451 } }
I20260808 04:05:24.044349 20745 heartbeater.cc:503] Master 127.5.83.62:35223 was elected leader, sending a full tablet report...
I20260808 04:05:24.044545 20980 raft_consensus.cc:348] T 8b5ab39a877f42569cfed68d4a9c324b P 9b42bba36bc44ed6a2f77dfe1716befb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efc6cb40fe94d259fdddadf6d4f7bc8" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 46599 } } peers { permanent_uuid: "9b42bba36bc44ed6a2f77dfe1716befb" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 35651 } } peers { permanent_uuid: "aae4238078a346ffb86fddbf8b4da39f" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 40451 } }
I20260808 04:05:24.044922 20980 raft_consensus.cc:374] T 8b5ab39a877f42569cfed68d4a9c324b P 9b42bba36bc44ed6a2f77dfe1716befb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:05:24.044996 20979 ts_tablet_manager.cc:1428] T 8b5ab39a877f42569cfed68d4a9c324b P aae4238078a346ffb86fddbf8b4da39f: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260808 04:05:24.045014 20980 raft_consensus.cc:729] T 8b5ab39a877f42569cfed68d4a9c324b P 9b42bba36bc44ed6a2f77dfe1716befb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b42bba36bc44ed6a2f77dfe1716befb, State: Initialized, Role: FOLLOWER
I20260808 04:05:24.045627 20980 consensus_queue.cc:260] T 8b5ab39a877f42569cfed68d4a9c324b P 9b42bba36bc44ed6a2f77dfe1716befb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efc6cb40fe94d259fdddadf6d4f7bc8" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 46599 } } peers { permanent_uuid: "9b42bba36bc44ed6a2f77dfe1716befb" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 35651 } } peers { permanent_uuid: "aae4238078a346ffb86fddbf8b4da39f" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 40451 } }
I20260808 04:05:24.047298 20980 ts_tablet_manager.cc:1428] T 8b5ab39a877f42569cfed68d4a9c324b P 9b42bba36bc44ed6a2f77dfe1716befb: Time spent starting tablet: real 0.013s user 0.011s sys 0.000s
W20260808 04:05:24.174845 20956 tablet.cc:2369] T 8b5ab39a877f42569cfed68d4a9c324b P 6efc6cb40fe94d259fdddadf6d4f7bc8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 04:05:24.189057 20746 tablet.cc:2369] T 8b5ab39a877f42569cfed68d4a9c324b P aae4238078a346ffb86fddbf8b4da39f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 04:05:24.195962 20547 tablet.cc:2369] T 8b5ab39a877f42569cfed68d4a9c324b P 9b42bba36bc44ed6a2f77dfe1716befb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:05:24.212344 20990 raft_consensus.cc:482] T 8b5ab39a877f42569cfed68d4a9c324b P 9b42bba36bc44ed6a2f77dfe1716befb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:05:24.212651 20990 raft_consensus.cc:504] T 8b5ab39a877f42569cfed68d4a9c324b P 9b42bba36bc44ed6a2f77dfe1716befb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efc6cb40fe94d259fdddadf6d4f7bc8" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 46599 } } peers { permanent_uuid: "9b42bba36bc44ed6a2f77dfe1716befb" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 35651 } } peers { permanent_uuid: "aae4238078a346ffb86fddbf8b4da39f" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 40451 } }
I20260808 04:05:24.214354 20990 leader_election.cc:288] T 8b5ab39a877f42569cfed68d4a9c324b P 9b42bba36bc44ed6a2f77dfe1716befb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6efc6cb40fe94d259fdddadf6d4f7bc8 (127.5.83.3:46599), aae4238078a346ffb86fddbf8b4da39f (127.5.83.2:40451)
I20260808 04:05:24.221698 20971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:24.215194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.1:49699 (local address 127.5.83.2:40451)
0808 04:05:24.215427 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0808 04:05:24.215433 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:05:24.215778 (+ 345us) server_negotiation.cc:407] Connection header received
0808 04:05:24.216128 (+ 350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:05:24.216134 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:05:24.216222 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:05:24.216331 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:05:24.216771 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:24.217672 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:24.219841 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:24.220745 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:24.220864 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:24.220960 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0808 04:05:24.221297 (+ 337us) server_negotiation.cc:299] Negotiation successful
0808 04:05:24.221439 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260808 04:05:24.221763 20993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:24.214650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.2:40451 (local address 127.5.83.1:49699)
0808 04:05:24.215613 (+ 963us) negotiation.cc:107] Waiting for socket to connect
0808 04:05:24.215637 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0808 04:05:24.215912 (+ 275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:05:24.216339 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:05:24.216342 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:05:24.216363 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:05:24.216613 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:24.216621 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:24.217827 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:24.217832 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:24.219675 (+ 1843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:24.219688 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:24.220969 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:24.220976 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:24.221120 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:24.221154 (+ 34us) client_negotiation.cc:769] Sending connection context
0808 04:05:24.221251 (+ 97us) client_negotiation.cc:240] Negotiation successful
0808 04:05:24.221380 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":801,"mutex_wait_us":30,"thread_start_us":932,"threads_started":1}
I20260808 04:05:24.222802 20671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b5ab39a877f42569cfed68d4a9c324b" candidate_uuid: "9b42bba36bc44ed6a2f77dfe1716befb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae4238078a346ffb86fddbf8b4da39f" is_pre_election: true
I20260808 04:05:24.223328 20671 raft_consensus.cc:2454] T 8b5ab39a877f42569cfed68d4a9c324b P aae4238078a346ffb86fddbf8b4da39f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b42bba36bc44ed6a2f77dfe1716befb in term 0.
I20260808 04:05:24.223420 20968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:24.215601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.1:57913 (local address 127.5.83.3:46599)
0808 04:05:24.215817 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0808 04:05:24.215823 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:05:24.215969 (+ 146us) server_negotiation.cc:407] Connection header received
0808 04:05:24.216529 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:05:24.216535 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:05:24.216609 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:05:24.216695 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:05:24.217273 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:24.218432 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:24.221165 (+ 2733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:24.222293 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:24.222412 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:24.222515 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0808 04:05:24.223051 (+ 536us) server_negotiation.cc:299] Negotiation successful
0808 04:05:24.223177 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 04:05:24.223506 20992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:24.214789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.3:46599 (local address 127.5.83.1:57913)
0808 04:05:24.215881 (+ 1092us) negotiation.cc:107] Waiting for socket to connect
0808 04:05:24.215897 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:05:24.216004 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:05:24.216731 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:05:24.216735 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:05:24.216760 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:05:24.217119 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:24.217131 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:24.218617 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:24.218624 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:24.221000 (+ 2376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:05:24.221018 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:05:24.222664 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:05:24.222671 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:05:24.222869 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:24.222902 (+ 33us) client_negotiation.cc:769] Sending connection context
0808 04:05:24.223002 (+ 100us) client_negotiation.cc:240] Negotiation successful
0808 04:05:24.223132 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":982,"mutex_wait_us":26,"thread_start_us":405,"threads_started":1}
I20260808 04:05:24.224081 20375 leader_election.cc:302] T 8b5ab39a877f42569cfed68d4a9c324b P 9b42bba36bc44ed6a2f77dfe1716befb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b42bba36bc44ed6a2f77dfe1716befb, aae4238078a346ffb86fddbf8b4da39f; no voters:
I20260808 04:05:24.224532 20990 raft_consensus.cc:2781] T 8b5ab39a877f42569cfed68d4a9c324b P 9b42bba36bc44ed6a2f77dfe1716befb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:05:24.224656 20990 raft_consensus.cc:482] T 8b5ab39a877f42569cfed68d4a9c324b P 9b42bba36bc44ed6a2f77dfe1716befb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:05:24.224754 20990 raft_consensus.cc:3037] T 8b5ab39a877f42569cfed68d4a9c324b P 9b42bba36bc44ed6a2f77dfe1716befb [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:05:24.224622 20873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b5ab39a877f42569cfed68d4a9c324b" candidate_uuid: "9b42bba36bc44ed6a2f77dfe1716befb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6efc6cb40fe94d259fdddadf6d4f7bc8" is_pre_election: true
I20260808 04:05:24.225163 20873 raft_consensus.cc:2454] T 8b5ab39a877f42569cfed68d4a9c324b P 6efc6cb40fe94d259fdddadf6d4f7bc8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b42bba36bc44ed6a2f77dfe1716befb in term 0.
I20260808 04:05:24.227239 20990 raft_consensus.cc:504] T 8b5ab39a877f42569cfed68d4a9c324b P 9b42bba36bc44ed6a2f77dfe1716befb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efc6cb40fe94d259fdddadf6d4f7bc8" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 46599 } } peers { permanent_uuid: "9b42bba36bc44ed6a2f77dfe1716befb" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 35651 } } peers { permanent_uuid: "aae4238078a346ffb86fddbf8b4da39f" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 40451 } }
I20260808 04:05:24.227851 20990 leader_election.cc:288] T 8b5ab39a877f42569cfed68d4a9c324b P 9b42bba36bc44ed6a2f77dfe1716befb [CANDIDATE]: Term 1 election: Requested vote from peers 6efc6cb40fe94d259fdddadf6d4f7bc8 (127.5.83.3:46599), aae4238078a346ffb86fddbf8b4da39f (127.5.83.2:40451)
I20260808 04:05:24.228231 20873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b5ab39a877f42569cfed68d4a9c324b" candidate_uuid: "9b42bba36bc44ed6a2f77dfe1716befb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6efc6cb40fe94d259fdddadf6d4f7bc8"
I20260808 04:05:24.228375 20671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b5ab39a877f42569cfed68d4a9c324b" candidate_uuid: "9b42bba36bc44ed6a2f77dfe1716befb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae4238078a346ffb86fddbf8b4da39f"
I20260808 04:05:24.228434 20873 raft_consensus.cc:3037] T 8b5ab39a877f42569cfed68d4a9c324b P 6efc6cb40fe94d259fdddadf6d4f7bc8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:05:24.228576 20671 raft_consensus.cc:3037] T 8b5ab39a877f42569cfed68d4a9c324b P aae4238078a346ffb86fddbf8b4da39f [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:05:24.231570 20671 raft_consensus.cc:2454] T 8b5ab39a877f42569cfed68d4a9c324b P aae4238078a346ffb86fddbf8b4da39f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b42bba36bc44ed6a2f77dfe1716befb in term 1.
I20260808 04:05:24.231575 20873 raft_consensus.cc:2454] T 8b5ab39a877f42569cfed68d4a9c324b P 6efc6cb40fe94d259fdddadf6d4f7bc8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b42bba36bc44ed6a2f77dfe1716befb in term 1.
I20260808 04:05:24.232110 20375 leader_election.cc:302] T 8b5ab39a877f42569cfed68d4a9c324b P 9b42bba36bc44ed6a2f77dfe1716befb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b42bba36bc44ed6a2f77dfe1716befb, aae4238078a346ffb86fddbf8b4da39f; no voters:
I20260808 04:05:24.232414 20990 raft_consensus.cc:2781] T 8b5ab39a877f42569cfed68d4a9c324b P 9b42bba36bc44ed6a2f77dfe1716befb [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:05:24.232887 20990 raft_consensus.cc:686] T 8b5ab39a877f42569cfed68d4a9c324b P 9b42bba36bc44ed6a2f77dfe1716befb [term 1 LEADER]: Becoming Leader. State: Replica: 9b42bba36bc44ed6a2f77dfe1716befb, State: Running, Role: LEADER
I20260808 04:05:24.233361 20990 consensus_queue.cc:237] T 8b5ab39a877f42569cfed68d4a9c324b P 9b42bba36bc44ed6a2f77dfe1716befb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efc6cb40fe94d259fdddadf6d4f7bc8" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 46599 } } peers { permanent_uuid: "9b42bba36bc44ed6a2f77dfe1716befb" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 35651 } } peers { permanent_uuid: "aae4238078a346ffb86fddbf8b4da39f" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 40451 } }
I20260808 04:05:24.238145 20244 catalog_manager.cc:5495] T 8b5ab39a877f42569cfed68d4a9c324b P 9b42bba36bc44ed6a2f77dfe1716befb reported cstate change: term changed from 0 to 1, leader changed from <none> to 9b42bba36bc44ed6a2f77dfe1716befb (127.5.83.1). New cstate: current_term: 1 leader_uuid: "9b42bba36bc44ed6a2f77dfe1716befb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efc6cb40fe94d259fdddadf6d4f7bc8" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 46599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b42bba36bc44ed6a2f77dfe1716befb" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 35651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aae4238078a346ffb86fddbf8b4da39f" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 40451 } health_report { overall_health: UNKNOWN } } }
I20260808 04:05:24.289567 20973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:24.283354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59558 (local address 127.5.83.1:35651)
0808 04:05:24.283749 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0808 04:05:24.283756 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:05:24.283776 (+ 20us) server_negotiation.cc:407] Connection header received
0808 04:05:24.283891 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:05:24.283896 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:05:24.284020 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:05:24.284196 (+ 176us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:05:24.284517 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:24.285650 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:24.286940 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:24.287623 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:24.287791 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:24.288245 (+ 454us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:05:24.288939 (+ 694us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:05:24.289085 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0808 04:05:24.289180 (+ 95us) server_negotiation.cc:299] Negotiation successful
0808 04:05:24.289267 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207}
I20260808 04:05:24.298472 20671 raft_consensus.cc:1261] T 8b5ab39a877f42569cfed68d4a9c324b P aae4238078a346ffb86fddbf8b4da39f [term 1 FOLLOWER]: Refusing update from remote peer 9b42bba36bc44ed6a2f77dfe1716befb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:05:24.298636 20873 raft_consensus.cc:1261] T 8b5ab39a877f42569cfed68d4a9c324b P 6efc6cb40fe94d259fdddadf6d4f7bc8 [term 1 FOLLOWER]: Refusing update from remote peer 9b42bba36bc44ed6a2f77dfe1716befb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:05:24.299449 20994 consensus_queue.cc:1035] T 8b5ab39a877f42569cfed68d4a9c324b P 9b42bba36bc44ed6a2f77dfe1716befb [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae4238078a346ffb86fddbf8b4da39f" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 40451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:05:24.299814 20990 consensus_queue.cc:1035] T 8b5ab39a877f42569cfed68d4a9c324b P 9b42bba36bc44ed6a2f77dfe1716befb [LEADER]: Connected to new peer: Peer: permanent_uuid: "6efc6cb40fe94d259fdddadf6d4f7bc8" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 46599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:05:24.314899 20998 mvcc.cc:205] Tried to move back new op lower bound from 7316119241916899328 to 7316119241663123456. Current Snapshot: MvccSnapshot[applied={T|T < 7316119241916899328}]
I20260808 04:05:24.340301 20244 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42892:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:05:24.340587 20244 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42892:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:05:24.344194 20244 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6e60926fd0f14f59a77f63e7f8ffcccb: Sending DeleteTablet for 3 replicas of tablet 8b5ab39a877f42569cfed68d4a9c324b
I20260808 04:05:24.345353 20841 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b5ab39a877f42569cfed68d4a9c324b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:05:24 UTC) from {username='kudu', principal='kudu/127.5.83.62@KRBTEST.COM'} at 127.0.0.1:36540
I20260808 04:05:24.345656 20642 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b5ab39a877f42569cfed68d4a9c324b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:05:24 UTC) from {username='kudu', principal='kudu/127.5.83.62@KRBTEST.COM'} at 127.0.0.1:40760
I20260808 04:05:24.345788 20445 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b5ab39a877f42569cfed68d4a9c324b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:05:24 UTC) from {username='kudu', principal='kudu/127.5.83.62@KRBTEST.COM'} at 127.0.0.1:59544
I20260808 04:05:24.347391 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 20349
I20260808 04:05:24.364392 21018 tablet_replica.cc:330] T 8b5ab39a877f42569cfed68d4a9c324b P 6efc6cb40fe94d259fdddadf6d4f7bc8: stopping tablet replica
I20260808 04:05:24.364939 21020 tablet_replica.cc:330] T 8b5ab39a877f42569cfed68d4a9c324b P 9b42bba36bc44ed6a2f77dfe1716befb: stopping tablet replica
I20260808 04:05:24.365505 21019 tablet_replica.cc:330] T 8b5ab39a877f42569cfed68d4a9c324b P aae4238078a346ffb86fddbf8b4da39f: stopping tablet replica
I20260808 04:05:24.365819 21018 raft_consensus.cc:2229] T 8b5ab39a877f42569cfed68d4a9c324b P 6efc6cb40fe94d259fdddadf6d4f7bc8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:05:24.366302 21019 raft_consensus.cc:2229] T 8b5ab39a877f42569cfed68d4a9c324b P aae4238078a346ffb86fddbf8b4da39f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:05:24.366364 21018 raft_consensus.cc:2258] T 8b5ab39a877f42569cfed68d4a9c324b P 6efc6cb40fe94d259fdddadf6d4f7bc8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:05:24.366495 21020 raft_consensus.cc:2229] T 8b5ab39a877f42569cfed68d4a9c324b P 9b42bba36bc44ed6a2f77dfe1716befb [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:05:24.367017 21019 raft_consensus.cc:2258] T 8b5ab39a877f42569cfed68d4a9c324b P aae4238078a346ffb86fddbf8b4da39f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:05:24.367659 21020 raft_consensus.cc:2258] T 8b5ab39a877f42569cfed68d4a9c324b P 9b42bba36bc44ed6a2f77dfe1716befb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:05:24.368880 21018 ts_tablet_manager.cc:1905] T 8b5ab39a877f42569cfed68d4a9c324b P 6efc6cb40fe94d259fdddadf6d4f7bc8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:05:24.370718 21020 ts_tablet_manager.cc:1905] T 8b5ab39a877f42569cfed68d4a9c324b P 9b42bba36bc44ed6a2f77dfe1716befb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:05:24.371850 21019 ts_tablet_manager.cc:1905] T 8b5ab39a877f42569cfed68d4a9c324b P aae4238078a346ffb86fddbf8b4da39f: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 08 04:05:24 dist-test-slave-1l3l krb5kdc[20179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.1@KRBTEST.COM
I20260808 04:05:24.376536 21018 ts_tablet_manager.cc:1918] T 8b5ab39a877f42569cfed68d4a9c324b P 6efc6cb40fe94d259fdddadf6d4f7bc8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:05:24.376717 21018 log.cc:1196] T 8b5ab39a877f42569cfed68d4a9c324b P 6efc6cb40fe94d259fdddadf6d4f7bc8: Deleting WAL directory at /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/minicluster-data/ts-2/wal/wals/8b5ab39a877f42569cfed68d4a9c324b
I20260808 04:05:24.377198 21018 ts_tablet_manager.cc:1939] T 8b5ab39a877f42569cfed68d4a9c324b P 6efc6cb40fe94d259fdddadf6d4f7bc8: Deleting consensus metadata
I20260808 04:05:24.378947 20229 catalog_manager.cc:4841] TS 6efc6cb40fe94d259fdddadf6d4f7bc8 (127.5.83.3:46599): tablet 8b5ab39a877f42569cfed68d4a9c324b (table test-table [id=52a97a249f9141ff87bd60f7f33c95b6]) successfully deleted
I20260808 04:05:24.379729 21019 ts_tablet_manager.cc:1918] T 8b5ab39a877f42569cfed68d4a9c324b P aae4238078a346ffb86fddbf8b4da39f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:05:24.380012 21019 log.cc:1196] T 8b5ab39a877f42569cfed68d4a9c324b P aae4238078a346ffb86fddbf8b4da39f: Deleting WAL directory at /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161826605655-5452-0/minicluster-data/ts-1/wal/wals/8b5ab39a877f42569cfed68d4a9c324b
I20260808 04:05:24.380580 21019 ts_tablet_manager.cc:1939] T 8b5ab39a877f42569cfed68d4a9c324b P aae4238078a346ffb86fddbf8b4da39f: Deleting consensus metadata
I20260808 04:05:24.381613 20973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:24.350384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59572 (local address 127.5.83.1:35651)
0808 04:05:24.350555 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0808 04:05:24.350561 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:05:24.356130 (+ 5569us) server_negotiation.cc:407] Connection header received
0808 04:05:24.356410 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:05:24.356417 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:05:24.356507 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:05:24.356579 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:05:24.358361 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:24.359623 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:24.366921 (+ 7298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:24.367678 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:24.367806 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:24.373036 (+ 5230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:05:24.373048 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:05:24.373063 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:05:24.373123 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:05:24.376713 (+ 3590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:05:24.377500 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:05:24.377508 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:05:24.377512 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:05:24.377669 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:05:24.378772 (+ 1103us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:05:24.378777 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:05:24.378779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:05:24.379221 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:05:24.379542 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0808 04:05:24.381009 (+ 1467us) server_negotiation.cc:299] Negotiation successful
0808 04:05:24.381272 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 04:05:24.382314 20223 catalog_manager.cc:4841] TS aae4238078a346ffb86fddbf8b4da39f (127.5.83.2:40451): tablet 8b5ab39a877f42569cfed68d4a9c324b (table test-table [id=52a97a249f9141ff87bd60f7f33c95b6]) successfully deleted
I20260808 04:05:24.383096 20540 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:05:24.480145 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 20349
W20260808 04:05:24.504009 20229 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.83.1:35651 (error 108)
I20260808 04:05:24.504220 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 20552
I20260808 04:05:24.506199 21028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:24.505553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:35651 (local address 127.0.0.1:59582)
0808 04:05:24.505918 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0808 04:05:24.506017 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.1:35651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":187,"threads_started":1}
W20260808 04:05:24.506418 20229 catalog_manager.cc:4584] TS 9b42bba36bc44ed6a2f77dfe1716befb (127.5.83.1:35651): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8b5ab39a877f42569cfed68d4a9c324b: Network error: Client connection negotiation failed: client connection to 127.5.83.1:35651: connect: Connection refused (error 111)
Aug 08 04:05:24 dist-test-slave-1l3l krb5kdc[20179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.2@KRBTEST.COM
I20260808 04:05:24.518319 20971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:24.505016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40762 (local address 127.5.83.2:40451)
0808 04:05:24.505208 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0808 04:05:24.505212 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:05:24.505231 (+ 19us) server_negotiation.cc:407] Connection header received
0808 04:05:24.505365 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:05:24.505370 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:05:24.505500 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:05:24.505619 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:05:24.507154 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:24.507948 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:24.509120 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:24.509502 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:24.509593 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:24.513268 (+ 3675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:05:24.513281 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:05:24.513295 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:05:24.513326 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:05:24.515990 (+ 2664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:05:24.516577 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:05:24.516583 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:05:24.516585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:05:24.516694 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:05:24.516967 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:05:24.516970 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:05:24.516972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:05:24.517329 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:05:24.517551 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0808 04:05:24.517855 (+ 304us) server_negotiation.cc:299] Negotiation successful
0808 04:05:24.518005 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260808 04:05:24.519018 20734 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:05:24.560001 21028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:24.559541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:35651 (local address 127.0.0.1:59588)
0808 04:05:24.559769 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0808 04:05:24.559859 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.1:35651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260808 04:05:24.560945 21028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:24.560595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:35651 (local address 127.0.0.1:59600)
0808 04:05:24.560738 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0808 04:05:24.560815 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.1:35651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260808 04:05:24.608278 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 20552
I20260808 04:05:24.621196 21028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:24.620747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:35651 (local address 127.0.0.1:59612)
0808 04:05:24.620937 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0808 04:05:24.621049 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.1:35651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260808 04:05:24.622156 21028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:24.621799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:35651 (local address 127.0.0.1:59620)
0808 04:05:24.621940 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0808 04:05:24.622010 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.1:35651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260808 04:05:24.633524 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 20752
Aug 08 04:05:24 dist-test-slave-1l3l krb5kdc[20179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.3@KRBTEST.COM
I20260808 04:05:24.647614 20968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:24.634252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36542 (local address 127.5.83.3:46599)
0808 04:05:24.634422 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0808 04:05:24.634427 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:05:24.634445 (+ 18us) server_negotiation.cc:407] Connection header received
0808 04:05:24.634576 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:05:24.634579 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:05:24.634654 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:05:24.634735 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:05:24.636043 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:24.636775 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:24.637946 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:05:24.638337 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:05:24.638436 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:05:24.642658 (+ 4222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:05:24.642673 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:05:24.642689 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:05:24.642723 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:05:24.645368 (+ 2645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:05:24.645969 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:05:24.645977 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:05:24.645979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:05:24.646117 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:05:24.646392 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:05:24.646395 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:05:24.646397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:05:24.646727 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:05:24.646941 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0808 04:05:24.647162 (+ 221us) server_negotiation.cc:299] Negotiation successful
0808 04:05:24.647369 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 04:05:24.648370 20947 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:05:24.702425 21028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:24.701973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:35651 (local address 127.0.0.1:59632)
0808 04:05:24.702172 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0808 04:05:24.702270 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.1:35651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260808 04:05:24.703507 21028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:05:24.703087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:35651 (local address 127.0.0.1:59638)
0808 04:05:24.703293 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0808 04:05:24.703373 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.1:35651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260808 04:05:24.734884 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 20752
I20260808 04:05:24.755683 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 20199
I20260808 04:05:24.756865 20306 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:05:24.864004 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 20199
2026-08-08T04:05:24Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4341 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:05:24 dist-test-slave-1l3l krb5kdc[21036](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:05:24 dist-test-slave-1l3l krb5kdc[21036](info): set up 2 sockets
Aug 08 04:05:24 dist-test-slave-1l3l krb5kdc[21036](info): commencing operation
krb5kdc: starting...
W20260808 04:05:26.053481 5452 mini_kdc.cc:121] Time spent starting KDC: real 1.143s user 0.000s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/krb5kdc/test-user.keytab.
Aug 08 04:05:26 dist-test-slave-1l3l krb5kdc[21036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:05:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:05:26Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.5.83.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.5.83.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.5.83.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/krb5kdc/rangeradmin_127.5.83.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.5.83.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/krb5kdc/rangeradmin_127.5.83.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.5.83.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.5.83.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.5.83.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/krb5kdc/rangerlookup_127.5.83.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.5.83.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/krb5kdc/rangerlookup_127.5.83.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.5.83.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.5.83.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.83.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/krb5kdc/HTTP_127.5.83.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.5.83.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/krb5kdc/HTTP_127.5.83.20@KRBTEST.COM.keytab.
I20260808 04:05:26.377082 5452 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task2cd2tT/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/postgres -l logfile start
2026-08-08 04:05:28.632 UTC [21081] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-08 04:05:28.633 UTC [21081] LOG: listening on IPv4 address "127.5.83.20", port 41723
2026-08-08 04:05:28.636 UTC [21081] LOG: listening on Unix socket "/tmp/.s.PGSQL.41723"
2026-08-08 04:05:28.657 UTC [21087] LOG: database system was shut down at 2026-08-08 04:05:27 UTC
2026-08-08 04:05:28.660 UTC [21081] LOG: database system is ready to accept connections
I20260808 04:05:29.722062 5452 mini_postgres.cc:96] Postgres bound to 41723
2026-08-08 04:05:29.744 UTC [21097] FATAL: database "slave" does not exist
127.5.83.20:41723 - accepting connections
I20260808 04:05:29.745524 5452 mini_ranger.cc:161] Starting Ranger...
I20260808 04:05:29.772112 5452 mini_ranger.cc:84] Created miniranger Postgres user
I20260808 04:05:30.167335 5452 mini_ranger.cc:87] Created ranger Postgres database
I20260808 04:05:30.167464 5452 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/ranger-admin
2026-08-08 04:05:30,548 [I] DB FLAVOR :POSTGRES
2026-08-08 04:05:30,549 [I] --------- Verifying Ranger DB connection ---------
2026-08-08 04:05:30,549 [I] Checking connection..
2026-08-08 04:05:30,549 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2cd2tT/build/asan/bin/postgresql.jar:/tmp/dist-test-task2cd2tT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.20:41723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-08 04:05:30,807 [I] Checking connection passed.
2026-08-08 04:05:30,807 [I] --------- Verifying version history table ---------
2026-08-08 04:05:30,807 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2cd2tT/build/asan/bin/postgresql.jar:/tmp/dist-test-task2cd2tT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.20:41723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 04:05:31,043 [I] Table x_db_version_h does not exist in database ranger
2026-08-08 04:05:31,043 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2cd2tT/build/asan/bin/postgresql.jar:/tmp/dist-test-task2cd2tT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.20:41723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 04:05:31,293 [I] Table x_db_version_h does not exist in database ranger
2026-08-08 04:05:31,293 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-08 04:05:31,293 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2cd2tT/build/asan/bin/postgresql.jar:/tmp/dist-test-task2cd2tT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.20:41723/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task2cd2tT/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-08 04:05:31.513 UTC [21198] WARNING: there is no transaction in progress
2026-08-08 04:05:31,520 [I] create_dbversion_catalog.sql file imported successfully
2026-08-08 04:05:31,520 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2cd2tT/build/asan/bin/postgresql.jar:/tmp/dist-test-task2cd2tT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.20:41723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 04:05:31,760 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-08 04:05:31,760 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-08 04:05:31,760 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2cd2tT/build/asan/bin/postgresql.jar:/tmp/dist-test-task2cd2tT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.20:41723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-08 04:05:32,010 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2cd2tT/build/asan/bin/postgresql.jar:/tmp/dist-test-task2cd2tT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.20:41723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-08 04:05:32,239 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2cd2tT/build/asan/bin/postgresql.jar:/tmp/dist-test-task2cd2tT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.20:41723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-1l3l.c.gcp-upstream.internal','N') ;"
2026-08-08 04:05:32,485 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2cd2tT/build/asan/bin/postgresql.jar:/tmp/dist-test-task2cd2tT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.20:41723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-08 04:05:32,730 [I] Table x_portal_user does not exist in database ranger
2026-08-08 04:05:32,730 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2cd2tT/build/asan/bin/postgresql.jar:/tmp/dist-test-task2cd2tT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.20:41723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-08 04:05:32,968 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-08 04:05:32,968 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2cd2tT/build/asan/bin/postgresql.jar:/tmp/dist-test-task2cd2tT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.20:41723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-08 04:05:33,218 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2cd2tT/build/asan/bin/postgresql.jar:/tmp/dist-test-task2cd2tT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.20:41723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-08 04:05:33,459 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-08 04:05:33,459 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2cd2tT/build/asan/bin/postgresql.jar:/tmp/dist-test-task2cd2tT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.20:41723/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task2cd2tT/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-08 04:05:34.334 UTC [21362] WARNING: there is no transaction in progress
2026-08-08 04:05:34.348 UTC [21362] WARNING: there is no transaction in progress
2026-08-08 04:05:34.362 UTC [21362] WARNING: there is no transaction in progress
2026-08-08 04:05:34.375 UTC [21362] WARNING: there is no transaction in progress
2026-08-08 04:05:34.390 UTC [21362] WARNING: there is no transaction in progress
2026-08-08 04:05:34.405 UTC [21362] WARNING: there is no transaction in progress
2026-08-08 04:05:34.454 UTC [21362] WARNING: there is no transaction in progress
2026-08-08 04:05:34.460 UTC [21362] WARNING: there is no transaction in progress
2026-08-08 04:05:34.473 UTC [21362] WARNING: there is no transaction in progress
2026-08-08 04:05:34.483 UTC [21362] WARNING: there is no transaction in progress
2026-08-08 04:05:34.492 UTC [21362] WARNING: there is no transaction in progress
2026-08-08 04:05:34.504 UTC [21362] WARNING: there is no transaction in progress
2026-08-08 04:05:34.513 UTC [21362] WARNING: there is no transaction in progress
2026-08-08 04:05:34.519 UTC [21362] WARNING: there is no transaction in progress
2026-08-08 04:05:34.525 UTC [21362] WARNING: there is no transaction in progress
2026-08-08 04:05:34.530 UTC [21362] WARNING: there is no transaction in progress
2026-08-08 04:05:35,070 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-08 04:05:35,070 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2cd2tT/build/asan/bin/postgresql.jar:/tmp/dist-test-task2cd2tT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.20:41723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-08 04:05:35,317 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-08 04:05:35,317 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2cd2tT/build/asan/bin/postgresql.jar:/tmp/dist-test-task2cd2tT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.20:41723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-08 04:05:35,548 [I] Table x_portal_user already exists in database 'ranger'
2026-08-08 04:05:35,548 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2cd2tT/build/asan/bin/postgresql.jar:/tmp/dist-test-task2cd2tT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.20:41723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-08 04:05:35,790 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-08 04:05:35,790 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2cd2tT/build/asan/bin/postgresql.jar:/tmp/dist-test-task2cd2tT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.20:41723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-08 04:05:36,038 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2cd2tT/build/asan/bin/postgresql.jar:/tmp/dist-test-task2cd2tT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.20:41723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-08 04:05:36,286 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2cd2tT/build/asan/bin/postgresql.jar:/tmp/dist-test-task2cd2tT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.20:41723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-1l3l.c.gcp-upstream.internal';"
2026-08-08 04:05:36,515 [I] CORE_DB_SCHEMA import status has been updated
2026-08-08 04:05:36,515 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2cd2tT/build/asan/bin/postgresql.jar:/tmp/dist-test-task2cd2tT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.20:41723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-08 04:05:36,774 [I] DB_PATCHES have already been applied
I20260808 04:05:36.780128 5452 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/ranger-admin:/tmp/dist-test-task2cd2tT/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task2cd2tT/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task2cd2tT/build/asan/bin/postgresql.jar:/tmp/dist-test-task2cd2tT/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260808 04:05:36.780222 5452 mini_ranger.cc:193] Using host: 127.5.83.20
I20260808 04:05:36.799346 5452 mini_ranger.cc:227] Ranger admin URL: http://127.5.83.20:39107
Aug 08, 2026 4:05:37 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 08, 2026 4:05:37 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 08, 2026 4:05:37 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/ranger-admin/ews/webapp
Aug 08, 2026 4:05:37 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 08, 2026 4:05:37 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 08, 2026 4:05:37 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/ranger-admin/ews/webapp].
Aug 08, 2026 4:05:37 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260808 04:05:37.979984 5452 mini_ranger.cc:160] Time spent starting Ranger: real 8.234s user 0.007s sys 0.046s
Aug 08, 2026 4:05:56 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 08, 2026 4:05:56 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 08, 2026 4:05:56 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 08, 2026 4:05:56 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 08, 2026 4:05:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 08, 2026 4:05:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 08, 2026 4:05:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 08, 2026 4:05:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 08, 2026 4:05:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 08, 2026 4:05:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 08, 2026 4:05:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 08, 2026 4:05:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 08, 2026 4:05:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 08, 2026 4:05:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 08, 2026 4:05:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 08, 2026 4:05:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 08, 2026 4:05:56 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 08, 2026 4:05:57 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260808 04:05:58.069955 5452 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.5.83.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.5.83.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.5.83.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/krb5kdc/rangerkms_127.5.83.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.5.83.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/krb5kdc/rangerkms_127.5.83.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.5.83.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.5.83.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.83.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/krb5kdc/HTTP_127.5.83.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.5.83.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/krb5kdc/HTTP_127.5.83.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 08 04:05:58 dist-test-slave-1l3l krb5kdc[21036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161958, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260808 04:05:58.259131 5452 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260808 04:05:58.280473 5452 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260808 04:05:59.061800 5452 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260808 04:05:59.062114 5452 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/ranger-kms
I20260808 04:05:59.062252 5452 mini_ranger_kms.cc:227] Using postgres at 127.5.83.20:41723
2026-08-08 04:05:59,646 [I] DB FLAVOR :POSTGRES
2026-08-08 04:05:59,646 [I] --------- Verifying Ranger DB connection ---------
2026-08-08 04:05:59,646 [I] Checking connection
2026-08-08 04:05:59,646 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2cd2tT/build/asan/bin/postgresql.jar:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.20:41723/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-08 04:05:59,902 [I] connection success
2026-08-08 04:05:59,903 [I] --------- Verifying Ranger DB tables ---------
2026-08-08 04:05:59,903 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-08 04:05:59,903 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2cd2tT/build/asan/bin/postgresql.jar:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.20:41723/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-08 04:06:00,150 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-08 04:06:00,151 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-08 04:06:00,151 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-08 04:06:00,151 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2cd2tT/build/asan/bin/postgresql.jar:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.20:41723/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-08 04:06:00,425 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260808 04:06:00.851740 5452 mini_ranger_kms.cc:310] Created kms service
I20260808 04:06:01.048187 5452 mini_ranger_kms.cc:326] Created kudu user
I20260808 04:06:01.090968 5452 mini_ranger_kms.cc:343] Created rangerkms user
Aug 08 04:06:01 dist-test-slave-1l3l krb5kdc[21036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161958, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.5.83.20@KRBTEST.COM
I20260808 04:06:01.471303 5452 mini_ranger_kms.cc:384] Added ranger policy
I20260808 04:06:01.471627 5452 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/ranger-kms:/tmp/dist-test-task2cd2tT/build/asan/bin/postgresql.jar:/tmp/dist-test-task2cd2tT/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task2cd2tT/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260808 04:06:01.471688 5452 mini_ranger_kms.cc:240] Using host: 127.5.83.20
I20260808 04:06:01.500330 5452 mini_ranger_kms.cc:276] Ranger KMS PID: 21644
I20260808 04:06:01.500491 5452 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.5.83.20:60397
Aug 08, 2026 4:06:02 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 08, 2026 4:06:02 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 08, 2026 4:06:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 08, 2026 4:06:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 08, 2026 4:06:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/ranger-kms/ews/webapp].
Aug 08, 2026 4:06:02 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 08 04:06:03 dist-test-slave-1l3l krb5kdc[21036](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786161963, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.83.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:06:03.938246 5452 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.679s user 0.214s sys 0.400s
I20260808 04:06:03.938549 5452 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260808 04:06:03.938764 5452 mini_ranger_kms.cc:401] 127.5.83.20:60397/kms/v1/keys
Aug 08 04:06:04 dist-test-slave-1l3l krb5kdc[21036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161963, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.83.20@KRBTEST.COM for HTTP/127.5.83.20@KRBTEST.COM
Aug 08 04:06:04 dist-test-slave-1l3l krb5kdc[21036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161963, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.83.20@KRBTEST.COM for HTTP/127.5.83.20@KRBTEST.COM
Aug 08 04:06:06 dist-test-slave-1l3l krb5kdc[21036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.5.83.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab.
Entry for principal kudu/127.5.83.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab.
I20260808 04:06:06.853611 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:32911
--webserver_interface=127.5.83.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab
--principal=kudu/127.5.83.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:39937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.62:32911
--ranger_config_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.83.20:60397/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:06:07.223743 21724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:06:07.224020 21724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:06:07.224152 21724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:06:07.231406 21724 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:06:07.231498 21724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:06:07.231570 21724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:06:07.231648 21724 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:06:07.231729 21724 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:06:07.241729 21724 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:39937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/master-0/wal
--ranger_kms_url=127.5.83.20:60397/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.5.83.62:32911
--ranger_config_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab
--principal=kudu/127.5.83.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:32911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:06:07.242897 21724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:06:07.244220 21724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:06:07.255990 21735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:06:07.256073 21730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:06:07.255965 21731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:06:07.256652 21724 server_base.cc:1034] running on GCE node
I20260808 04:06:07.257377 21724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:06:07.258659 21724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:06:07.259848 21724 hybrid_clock.cc:648] HybridClock initialized: now 1786161967259823 us; error 51 us; skew 500 ppm
I20260808 04:06:07.260170 21724 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:06:07 dist-test-slave-1l3l krb5kdc[21036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161967, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:06:07.263141 21724 init.cc:377] Logged in from keytab as kudu/127.5.83.62@KRBTEST.COM (short username kudu)
I20260808 04:06:07.264577 21724 webserver.cc:460] Webserver started at http://127.5.83.62:35511/ using document root <none> and password file <none>
I20260808 04:06:07.265147 21724 fs_manager.cc:359] Metadata directory not provided
I20260808 04:06:07.265252 21724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:06:07.265547 21724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 04:06:07 dist-test-slave-1l3l krb5kdc[21036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161967, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.62@KRBTEST.COM for HTTP/127.5.83.20@KRBTEST.COM
I20260808 04:06:07.314378 21724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/master-0/data/instance:
uuid: "cf6929c542314cd3a2bf112a7ce508d2"
format_stamp: "Formatted at 2026-08-08 04:06:07 on dist-test-slave-1l3l"
server_key: "10205209b7ff0caa6e455a52e6acefa4"
server_key_iv: "3fa79e29348df88c1cacc76961ff4590"
server_key_version: "kuduclusterkey@0"
I20260808 04:06:07.315176 21724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/master-0/wal/instance:
uuid: "cf6929c542314cd3a2bf112a7ce508d2"
format_stamp: "Formatted at 2026-08-08 04:06:07 on dist-test-slave-1l3l"
server_key: "10205209b7ff0caa6e455a52e6acefa4"
server_key_iv: "3fa79e29348df88c1cacc76961ff4590"
server_key_version: "kuduclusterkey@0"
I20260808 04:06:07.320544 21724 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260808 04:06:07.340104 21746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:06:07.341488 21724 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 04:06:07.341645 21724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/master-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/master-0/wal
uuid: "cf6929c542314cd3a2bf112a7ce508d2"
format_stamp: "Formatted at 2026-08-08 04:06:07 on dist-test-slave-1l3l"
server_key: "10205209b7ff0caa6e455a52e6acefa4"
server_key_iv: "3fa79e29348df88c1cacc76961ff4590"
server_key_version: "kuduclusterkey@0"
I20260808 04:06:07.341864 21724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:06:07.359758 21724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:06:07.363785 21724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:06:07.364076 21724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:06:07.383951 21724 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.62:32911
I20260808 04:06:07.383970 21812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.62:32911 every 8 connection(s)
I20260808 04:06:07.385488 21724 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
I20260808 04:06:07.389341 21813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:06:07.395025 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 21724
I20260808 04:06:07.395252 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/master-0/wal/instance
I20260808 04:06:07.396175 5452 external_mini_cluster.cc:1428] Setting key 3a0a78239dd52680446f7078cc86c58e
I20260808 04:06:07.400246 21813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf6929c542314cd3a2bf112a7ce508d2: Bootstrap starting.
I20260808 04:06:07.404547 21813 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf6929c542314cd3a2bf112a7ce508d2: Neither blocks nor log segments found. Creating new log.
I20260808 04:06:07.406004 21813 log.cc:824] T 00000000000000000000000000000000 P cf6929c542314cd3a2bf112a7ce508d2: Log is configured to *not* fsync() on all Append() calls
I20260808 04:06:07.415220 21813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf6929c542314cd3a2bf112a7ce508d2: No bootstrap required, opened a new log
Aug 08 04:06:07 dist-test-slave-1l3l krb5kdc[21036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:06:07.422631 21813 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf6929c542314cd3a2bf112a7ce508d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf6929c542314cd3a2bf112a7ce508d2" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 32911 } }
I20260808 04:06:07.422964 21813 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf6929c542314cd3a2bf112a7ce508d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:06:07.423071 21813 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf6929c542314cd3a2bf112a7ce508d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf6929c542314cd3a2bf112a7ce508d2, State: Initialized, Role: FOLLOWER
I20260808 04:06:07.423633 21813 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf6929c542314cd3a2bf112a7ce508d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf6929c542314cd3a2bf112a7ce508d2" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 32911 } }
I20260808 04:06:07.423910 21813 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf6929c542314cd3a2bf112a7ce508d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:06:07.424081 21813 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf6929c542314cd3a2bf112a7ce508d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:06:07.424243 21813 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf6929c542314cd3a2bf112a7ce508d2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:06:07.426798 21813 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf6929c542314cd3a2bf112a7ce508d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf6929c542314cd3a2bf112a7ce508d2" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 32911 } }
I20260808 04:06:07.427301 21813 leader_election.cc:302] T 00000000000000000000000000000000 P cf6929c542314cd3a2bf112a7ce508d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf6929c542314cd3a2bf112a7ce508d2; no voters:
I20260808 04:06:07.427971 21813 leader_election.cc:288] T 00000000000000000000000000000000 P cf6929c542314cd3a2bf112a7ce508d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:06:07.428144 21819 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf6929c542314cd3a2bf112a7ce508d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:06:07.429306 21819 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf6929c542314cd3a2bf112a7ce508d2 [term 1 LEADER]: Becoming Leader. State: Replica: cf6929c542314cd3a2bf112a7ce508d2, State: Running, Role: LEADER
I20260808 04:06:07.429929 21819 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf6929c542314cd3a2bf112a7ce508d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf6929c542314cd3a2bf112a7ce508d2" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 32911 } }
I20260808 04:06:07.430169 21813 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf6929c542314cd3a2bf112a7ce508d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:06:07.432250 21817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:06:07.398367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52450 (local address 127.5.83.62:32911)
0808 04:06:07.411820 (+ 13453us) server_negotiation.cc:206] Beginning negotiation
0808 04:06:07.411837 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 04:06:07.412221 (+ 384us) server_negotiation.cc:407] Connection header received
0808 04:06:07.414530 (+ 2309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:06:07.414552 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:06:07.414917 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:06:07.415252 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:06:07.416606 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:06:07.417946 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:06:07.419214 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:06:07.419956 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:06:07.420135 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:06:07.423870 (+ 3735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:06:07.423880 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:06:07.423901 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:06:07.423942 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:06:07.426975 (+ 3033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:06:07.427821 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:06:07.427830 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:06:07.427838 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:06:07.428034 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:06:07.428528 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:06:07.428533 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:06:07.428536 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:06:07.429498 (+ 962us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:06:07.429819 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0808 04:06:07.430332 (+ 513us) server_negotiation.cc:299] Negotiation successful
0808 04:06:07.430732 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13185,"thread_start_us":12955,"threads_started":1}
I20260808 04:06:07.437793 21813 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260808 04:06:07.437856 21820 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf6929c542314cd3a2bf112a7ce508d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf6929c542314cd3a2bf112a7ce508d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf6929c542314cd3a2bf112a7ce508d2" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 32911 } } }
I20260808 04:06:07.438654 21821 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf6929c542314cd3a2bf112a7ce508d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf6929c542314cd3a2bf112a7ce508d2. Latest consensus state: current_term: 1 leader_uuid: "cf6929c542314cd3a2bf112a7ce508d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf6929c542314cd3a2bf112a7ce508d2" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 32911 } } }
I20260808 04:06:07.438217 21820 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf6929c542314cd3a2bf112a7ce508d2 [sys.catalog]: This master's current role is: LEADER
I20260808 04:06:07.439002 21821 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf6929c542314cd3a2bf112a7ce508d2 [sys.catalog]: This master's current role is: LEADER
I20260808 04:06:07.440267 21848 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:06:07.453898 21848 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:06:07.464362 21848 catalog_manager.cc:1347] Generated new cluster ID: c7ae0f9cae0a424192ae3623bf908885
I20260808 04:06:07.464532 21848 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:06:07.488950 21848 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:06:07.490942 21848 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:06:07.513303 21848 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cf6929c542314cd3a2bf112a7ce508d2: Generated new TSK 0
I20260808 04:06:07.514755 21848 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.83.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.1.keytab.
Entry for principal kudu/127.5.83.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.1.keytab.
I20260808 04:06:08.254598 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.83.20:60397/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.1:0
--local_ip_for_outbound_sockets=127.5.83.1
--webserver_interface=127.5.83.1
--webserver_port=0
--tserver_master_addrs=127.5.83.62:32911
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.1.keytab
--principal=kudu/127.5.83.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:39937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 04:06:08 dist-test-slave-1l3l krb5kdc[21036](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786161968, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260808 04:06:08.670868 21890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:06:08.671262 21890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:06:08.671497 21890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:06:08.682689 21890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:06:08.682870 21890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:06:08.683081 21890 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.1
I20260808 04:06:08.698395 21890 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:39937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.5.83.20:60397/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.1.keytab
--principal=kudu/127.5.83.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.83.1
--webserver_port=0
--tserver_master_addrs=127.5.83.62:32911
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.1
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:06:08.700124 21890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:06:08.701859 21890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:06:08.727531 21902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:06:08.740507 21903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:06:08.742669 21906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:06:08.743458 21890 server_base.cc:1034] running on GCE node
I20260808 04:06:08.744194 21890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:06:08.745322 21890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:06:08.746636 21890 hybrid_clock.cc:648] HybridClock initialized: now 1786161968746522 us; error 90 us; skew 500 ppm
I20260808 04:06:08.746974 21890 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:06:08 dist-test-slave-1l3l krb5kdc[21036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161968, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:06:08.750303 21890 init.cc:377] Logged in from keytab as kudu/127.5.83.1@KRBTEST.COM (short username kudu)
I20260808 04:06:08.751915 21890 webserver.cc:460] Webserver started at http://127.5.83.1:36333/ using document root <none> and password file <none>
I20260808 04:06:08.752593 21890 fs_manager.cc:359] Metadata directory not provided
I20260808 04:06:08.752712 21890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:06:08.753022 21890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 04:06:08 dist-test-slave-1l3l krb5kdc[21036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161968, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.1@KRBTEST.COM for HTTP/127.5.83.20@KRBTEST.COM
I20260808 04:06:08.774741 21890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/ts-0/data/instance:
uuid: "847881a547e74ea297246c29704acb99"
format_stamp: "Formatted at 2026-08-08 04:06:08 on dist-test-slave-1l3l"
server_key: "4c0b88fbec3c3994caeba80a8694171d"
server_key_iv: "9f45162e601b8a16ca4750da901a9bc0"
server_key_version: "kuduclusterkey@0"
I20260808 04:06:08.777063 21890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/ts-0/wal/instance:
uuid: "847881a547e74ea297246c29704acb99"
format_stamp: "Formatted at 2026-08-08 04:06:08 on dist-test-slave-1l3l"
server_key: "4c0b88fbec3c3994caeba80a8694171d"
server_key_iv: "9f45162e601b8a16ca4750da901a9bc0"
server_key_version: "kuduclusterkey@0"
I20260808 04:06:08.784502 21890 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260808 04:06:08.809440 21917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:06:08.811308 21890 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260808 04:06:08.811501 21890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/ts-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/ts-0/wal
uuid: "847881a547e74ea297246c29704acb99"
format_stamp: "Formatted at 2026-08-08 04:06:08 on dist-test-slave-1l3l"
server_key: "4c0b88fbec3c3994caeba80a8694171d"
server_key_iv: "9f45162e601b8a16ca4750da901a9bc0"
server_key_version: "kuduclusterkey@0"
I20260808 04:06:08.811786 21890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:06:08.831733 21890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:06:08.837107 21890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:06:08.837479 21890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:06:08.844794 21890 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:06:08.847765 21890 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:06:08.847930 21890 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260808 04:06:08.848076 21890 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:06:08.848146 21890 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 04:06:09 dist-test-slave-1l3l krb5kdc[21036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161968, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.62@KRBTEST.COM for HTTP/127.5.83.20@KRBTEST.COM
I20260808 04:06:09.221962 21890 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.1:32869
I20260808 04:06:09.221992 22085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.1:32869 every 8 connection(s)
I20260808 04:06:09.223650 21890 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
I20260808 04:06:09.225477 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 21890
I20260808 04:06:09.225657 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/ts-0/wal/instance
I20260808 04:06:09.226454 5452 external_mini_cluster.cc:1428] Setting key 6621a2d1c61613bee0c18220acbe3d37
Aug 08 04:06:09 dist-test-slave-1l3l krb5kdc[21036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161968, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.1@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:06:09.251255 22090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:06:09.227333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:32911 (local address 127.5.83.1:56009)
0808 04:06:09.228302 (+ 969us) negotiation.cc:107] Waiting for socket to connect
0808 04:06:09.228363 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0808 04:06:09.230052 (+ 1689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:06:09.232593 (+ 2541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:06:09.232604 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:06:09.233221 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:06:09.234013 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:06:09.234038 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:06:09.235486 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:06:09.235491 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:06:09.237455 (+ 1964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:06:09.237479 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:06:09.238492 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:06:09.238499 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:06:09.238780 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:06:09.239559 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:06:09.239588 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:06:09.242274 (+ 2686us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:06:09.246495 (+ 4221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:06:09.246504 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:06:09.246522 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:06:09.247070 (+ 548us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:06:09.247500 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:06:09.247505 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:06:09.247508 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:06:09.247817 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:06:09.248456 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:06:09.248463 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:06:09.248844 (+ 381us) client_negotiation.cc:769] Sending connection context
0808 04:06:09.249121 (+ 277us) client_negotiation.cc:240] Negotiation successful
0808 04:06:09.249699 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":681,"thread_start_us":470,"threads_started":1}
I20260808 04:06:09.255895 22089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:06:09.226760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.1:56009 (local address 127.5.83.62:32911)
0808 04:06:09.232052 (+ 5292us) server_negotiation.cc:206] Beginning negotiation
0808 04:06:09.232057 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:06:09.232073 (+ 16us) server_negotiation.cc:407] Connection header received
0808 04:06:09.232124 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:06:09.232128 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:06:09.232189 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:06:09.232306 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:06:09.234227 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:06:09.235286 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:06:09.237610 (+ 2324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:06:09.238246 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:06:09.238380 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:06:09.242560 (+ 4180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:06:09.242567 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:06:09.242571 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:06:09.242626 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:06:09.246264 (+ 3638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:06:09.247210 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:06:09.247215 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:06:09.247219 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:06:09.247373 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:06:09.247945 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:06:09.247950 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:06:09.247953 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:06:09.248317 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:06:09.248481 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0808 04:06:09.255227 (+ 6746us) server_negotiation.cc:299] Negotiation successful
0808 04:06:09.255491 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5193,"thread_start_us":5083,"threads_started":1}
I20260808 04:06:09.257344 22086 heartbeater.cc:347] Connected to a master server at 127.5.83.62:32911
I20260808 04:06:09.257740 22086 heartbeater.cc:464] Registering TS with master...
I20260808 04:06:09.258630 22086 heartbeater.cc:511] Master 127.5.83.62:32911 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.5.83.2@KRBTEST.COM; defaulting to no policy
I20260808 04:06:09.261132 21767 ts_manager.cc:194] Registered new tserver with Master: 847881a547e74ea297246c29704acb99 (127.5.83.1:32869)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.2@KRBTEST.COM" created.
I20260808 04:06:09.263140 21767 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.1@KRBTEST.COM'} at 127.5.83.1:56009
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.2.keytab.
Entry for principal kudu/127.5.83.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.2.keytab.
I20260808 04:06:09.299110 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.83.20:60397/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.2:0
--local_ip_for_outbound_sockets=127.5.83.2
--webserver_interface=127.5.83.2
--webserver_port=0
--tserver_master_addrs=127.5.83.62:32911
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.2.keytab
--principal=kudu/127.5.83.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:39937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 04:06:09 dist-test-slave-1l3l krb5kdc[21036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161968, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.62@KRBTEST.COM for HTTP/127.5.83.20@KRBTEST.COM
W20260808 04:06:09.779186 22093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:06:09.779484 22093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:06:09.779646 22093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:06:09.787040 22093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:06:09.787159 22093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:06:09.787318 22093 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.2
I20260808 04:06:09.797277 22093 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:39937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.5.83.20:60397/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.2.keytab
--principal=kudu/127.5.83.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.83.2
--webserver_port=0
--tserver_master_addrs=127.5.83.62:32911
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.2
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:06:09.798568 22093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:06:09.799904 22093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:06:09.815454 22117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:06:09.816309 22118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:06:09.824890 22120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:06:09.826013 22093 server_base.cc:1034] running on GCE node
I20260808 04:06:09.826668 22093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:06:09.827690 22093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:06:09.829010 22093 hybrid_clock.cc:648] HybridClock initialized: now 1786161969828930 us; error 80 us; skew 500 ppm
I20260808 04:06:09.829324 22093 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:06:09 dist-test-slave-1l3l krb5kdc[21036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161969, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:06:09.832945 22093 init.cc:377] Logged in from keytab as kudu/127.5.83.2@KRBTEST.COM (short username kudu)
I20260808 04:06:09.836942 22093 webserver.cc:460] Webserver started at http://127.5.83.2:39359/ using document root <none> and password file <none>
I20260808 04:06:09.837652 22093 fs_manager.cc:359] Metadata directory not provided
I20260808 04:06:09.837771 22093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:06:09.838096 22093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 04:06:09 dist-test-slave-1l3l krb5kdc[21036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161969, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.2@KRBTEST.COM for HTTP/127.5.83.20@KRBTEST.COM
I20260808 04:06:09.855373 22093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/ts-1/data/instance:
uuid: "4d86eb79e0b34cfc8a4fc7e5f4049f1b"
format_stamp: "Formatted at 2026-08-08 04:06:09 on dist-test-slave-1l3l"
server_key: "38e342ec524214ba8d7f626f671cb218"
server_key_iv: "720cd6a9ff5f19a12bbf01d6a47dedc0"
server_key_version: "kuduclusterkey@0"
I20260808 04:06:09.856751 22093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/ts-1/wal/instance:
uuid: "4d86eb79e0b34cfc8a4fc7e5f4049f1b"
format_stamp: "Formatted at 2026-08-08 04:06:09 on dist-test-slave-1l3l"
server_key: "38e342ec524214ba8d7f626f671cb218"
server_key_iv: "720cd6a9ff5f19a12bbf01d6a47dedc0"
server_key_version: "kuduclusterkey@0"
I20260808 04:06:09.870208 22093 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.008s sys 0.004s
I20260808 04:06:09.910396 22132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:06:09.913188 22093 fs_manager.cc:725] Time spent opening block manager: real 0.019s user 0.006s sys 0.009s
I20260808 04:06:09.913476 22093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/ts-1/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/ts-1/wal
uuid: "4d86eb79e0b34cfc8a4fc7e5f4049f1b"
format_stamp: "Formatted at 2026-08-08 04:06:09 on dist-test-slave-1l3l"
server_key: "38e342ec524214ba8d7f626f671cb218"
server_key_iv: "720cd6a9ff5f19a12bbf01d6a47dedc0"
server_key_version: "kuduclusterkey@0"
I20260808 04:06:09.913807 22093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:06:09.941473 22093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:06:09.945467 22093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:06:09.945756 22093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:06:09.946712 22093 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:06:09.948217 22093 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:06:09.948303 22093 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:06:09.948438 22093 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:06:09.948493 22093 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:06:10.001986 22093 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.2:35155
I20260808 04:06:10.002003 22297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.2:35155 every 8 connection(s)
I20260808 04:06:10.003433 22093 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
I20260808 04:06:10.003750 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 22093
I20260808 04:06:10.003942 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/ts-1/wal/instance
I20260808 04:06:10.004549 5452 external_mini_cluster.cc:1428] Setting key 12c968c678683e90a75548454d369832
Aug 08 04:06:10 dist-test-slave-1l3l krb5kdc[21036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161969, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.2@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.83.3@KRBTEST.COM; defaulting to no policy
I20260808 04:06:10.046252 22301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:06:10.013843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.2:44087 (local address 127.5.83.62:32911)
0808 04:06:10.021304 (+ 7461us) server_negotiation.cc:206] Beginning negotiation
0808 04:06:10.021312 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:06:10.023864 (+ 2552us) server_negotiation.cc:407] Connection header received
0808 04:06:10.024869 (+ 1005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:06:10.024876 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:06:10.024968 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:06:10.025054 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:06:10.027427 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:06:10.028473 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:06:10.030854 (+ 2381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:06:10.031398 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:06:10.031506 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:06:10.036440 (+ 4934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:06:10.036447 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:06:10.036451 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:06:10.036492 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:06:10.039987 (+ 3495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:06:10.040726 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:06:10.040732 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:06:10.040735 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:06:10.040892 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:06:10.044290 (+ 3398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:06:10.044296 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:06:10.044301 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:06:10.044627 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:06:10.044733 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0808 04:06:10.045580 (+ 847us) server_negotiation.cc:299] Negotiation successful
0808 04:06:10.045877 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7321,"thread_start_us":7204,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.3@KRBTEST.COM" created.
I20260808 04:06:10.047427 22300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:06:10.013235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:32911 (local address 127.5.83.2:44087)
0808 04:06:10.022284 (+ 9049us) negotiation.cc:107] Waiting for socket to connect
0808 04:06:10.022349 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0808 04:06:10.024506 (+ 2157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:06:10.025608 (+ 1102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:06:10.025619 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:06:10.026337 (+ 718us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:06:10.027209 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:06:10.027234 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:06:10.028713 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:06:10.028719 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:06:10.030698 (+ 1979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:06:10.030713 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:06:10.031924 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:06:10.031933 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:06:10.032236 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:06:10.033332 (+ 1096us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:06:10.033366 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:06:10.036130 (+ 2764us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:06:10.040185 (+ 4055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:06:10.040191 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:06:10.040204 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:06:10.040601 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:06:10.043833 (+ 3232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:06:10.043839 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:06:10.043842 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:06:10.044146 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:06:10.044820 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:06:10.044826 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:06:10.045173 (+ 347us) client_negotiation.cc:769] Sending connection context
0808 04:06:10.045471 (+ 298us) client_negotiation.cc:240] Negotiation successful
0808 04:06:10.046005 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8759,"thread_start_us":8544,"threads_started":1}
I20260808 04:06:10.049365 22299 heartbeater.cc:347] Connected to a master server at 127.5.83.62:32911
I20260808 04:06:10.049697 22299 heartbeater.cc:464] Registering TS with master...
I20260808 04:06:10.050513 22299 heartbeater.cc:511] Master 127.5.83.62:32911 requested a full tablet report, sending...
I20260808 04:06:10.052244 21767 ts_manager.cc:194] Registered new tserver with Master: 4d86eb79e0b34cfc8a4fc7e5f4049f1b (127.5.83.2:35155)
I20260808 04:06:10.053261 21767 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.2@KRBTEST.COM'} at 127.5.83.2:44087
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.3.keytab.
Entry for principal kudu/127.5.83.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.3.keytab.
I20260808 04:06:10.078603 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.83.20:60397/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.3:0
--local_ip_for_outbound_sockets=127.5.83.3
--webserver_interface=127.5.83.3
--webserver_port=0
--tserver_master_addrs=127.5.83.62:32911
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.3.keytab
--principal=kudu/127.5.83.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:39937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:06:10.269636 22086 heartbeater.cc:503] Master 127.5.83.62:32911 was elected leader, sending a full tablet report...
W20260808 04:06:10.416111 22310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:06:10.416384 22310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:06:10.416527 22310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:06:10.423939 22310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:06:10.424034 22310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:06:10.424186 22310 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.3
I20260808 04:06:10.433636 22310 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:39937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.5.83.20:60397/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.3.keytab
--principal=kudu/127.5.83.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.83.3
--webserver_port=0
--tserver_master_addrs=127.5.83.62:32911
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.3
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:06:10.434825 22310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:06:10.436100 22310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:06:10.449910 22331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:06:10.449887 22332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:06:10.450599 22335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:06:10.450897 22310 server_base.cc:1034] running on GCE node
I20260808 04:06:10.451484 22310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:06:10.452389 22310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:06:10.453593 22310 hybrid_clock.cc:648] HybridClock initialized: now 1786161970453536 us; error 51 us; skew 500 ppm
I20260808 04:06:10.453876 22310 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:06:10 dist-test-slave-1l3l krb5kdc[21036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161970, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:06:10.457135 22310 init.cc:377] Logged in from keytab as kudu/127.5.83.3@KRBTEST.COM (short username kudu)
I20260808 04:06:10.458873 22310 webserver.cc:460] Webserver started at http://127.5.83.3:44123/ using document root <none> and password file <none>
I20260808 04:06:10.459510 22310 fs_manager.cc:359] Metadata directory not provided
I20260808 04:06:10.459627 22310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:06:10.459942 22310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 04:06:10 dist-test-slave-1l3l krb5kdc[21036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161970, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.3@KRBTEST.COM for HTTP/127.5.83.20@KRBTEST.COM
I20260808 04:06:10.476150 22310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/ts-2/data/instance:
uuid: "3e0c52ef668d46d3bcabf7a3e17417a2"
format_stamp: "Formatted at 2026-08-08 04:06:10 on dist-test-slave-1l3l"
server_key: "e9d7a41de25236a473d9b0f76cfb8faf"
server_key_iv: "d4d2f06f98dcec79d6ea5ae0f6ad82dd"
server_key_version: "kuduclusterkey@0"
I20260808 04:06:10.477166 22310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/ts-2/wal/instance:
uuid: "3e0c52ef668d46d3bcabf7a3e17417a2"
format_stamp: "Formatted at 2026-08-08 04:06:10 on dist-test-slave-1l3l"
server_key: "e9d7a41de25236a473d9b0f76cfb8faf"
server_key_iv: "d4d2f06f98dcec79d6ea5ae0f6ad82dd"
server_key_version: "kuduclusterkey@0"
I20260808 04:06:10.483747 22310 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.003s
I20260808 04:06:10.500371 22345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:06:10.502040 22310 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:06:10.502203 22310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/ts-2/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/ts-2/wal
uuid: "3e0c52ef668d46d3bcabf7a3e17417a2"
format_stamp: "Formatted at 2026-08-08 04:06:10 on dist-test-slave-1l3l"
server_key: "e9d7a41de25236a473d9b0f76cfb8faf"
server_key_iv: "d4d2f06f98dcec79d6ea5ae0f6ad82dd"
server_key_version: "kuduclusterkey@0"
I20260808 04:06:10.502424 22310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:06:10.536849 22310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:06:10.541276 22310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:06:10.541589 22310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:06:10.542671 22310 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:06:10.544291 22310 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:06:10.544390 22310 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:06:10.544493 22310 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:06:10.544548 22310 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:06:10.575883 22310 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.3:42495
I20260808 04:06:10.575906 22526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.3:42495 every 8 connection(s)
I20260808 04:06:10.577346 22310 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
I20260808 04:06:10.582626 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 22310
I20260808 04:06:10.582801 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/ts-2/wal/instance
I20260808 04:06:10.583351 5452 external_mini_cluster.cc:1428] Setting key c3fd8e37c8781c8e59f39add46d1a585
Aug 08 04:06:10 dist-test-slave-1l3l krb5kdc[21036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161970, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.3@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:06:10.598718 22529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:06:10.579934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.3:60091 (local address 127.5.83.62:32911)
0808 04:06:10.580416 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0808 04:06:10.580421 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:06:10.581422 (+ 1001us) server_negotiation.cc:407] Connection header received
0808 04:06:10.583676 (+ 2254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:06:10.583683 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:06:10.583804 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:06:10.583920 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:06:10.585751 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:06:10.586690 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:06:10.588842 (+ 2152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:06:10.589294 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:06:10.589390 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:06:10.593039 (+ 3649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:06:10.593044 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:06:10.593047 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:06:10.593080 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:06:10.595561 (+ 2481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:06:10.596280 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:06:10.596285 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:06:10.596287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:06:10.596408 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:06:10.596804 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:06:10.596808 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:06:10.596810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:06:10.597178 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:06:10.597311 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0808 04:06:10.598114 (+ 803us) server_negotiation.cc:299] Negotiation successful
0808 04:06:10.598340 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":241,"threads_started":1}
I20260808 04:06:10.599725 22530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:06:10.580491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:32911 (local address 127.5.83.3:60091)
0808 04:06:10.581254 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0808 04:06:10.581298 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 04:06:10.583249 (+ 1951us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:06:10.584204 (+ 955us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:06:10.584214 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:06:10.584789 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:06:10.585574 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:06:10.585597 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:06:10.586846 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:06:10.586851 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:06:10.588686 (+ 1835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:06:10.588711 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:06:10.589495 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:06:10.589501 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:06:10.589768 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:06:10.590446 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:06:10.590479 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:06:10.592828 (+ 2349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:06:10.595717 (+ 2889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:06:10.595727 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:06:10.595743 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:06:10.596157 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:06:10.596511 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:06:10.596516 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:06:10.596520 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:06:10.596692 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:06:10.597299 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:06:10.597306 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:06:10.597706 (+ 400us) client_negotiation.cc:769] Sending connection context
0808 04:06:10.598004 (+ 298us) client_negotiation.cc:240] Negotiation successful
0808 04:06:10.598505 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":332,"threads_started":1}
I20260808 04:06:10.600991 22527 heartbeater.cc:347] Connected to a master server at 127.5.83.62:32911
I20260808 04:06:10.601274 22527 heartbeater.cc:464] Registering TS with master...
I20260808 04:06:10.601943 22527 heartbeater.cc:511] Master 127.5.83.62:32911 requested a full tablet report, sending...
I20260808 04:06:10.603298 21767 ts_manager.cc:194] Registered new tserver with Master: 3e0c52ef668d46d3bcabf7a3e17417a2 (127.5.83.3:42495)
I20260808 04:06:10.604355 21767 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.3@KRBTEST.COM'} at 127.5.83.3:60091
I20260808 04:06:10.610875 5452 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:06:10.630633 22529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:06:10.616411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51308 (local address 127.5.83.62:32911)
0808 04:06:10.616600 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0808 04:06:10.616607 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:06:10.617142 (+ 535us) server_negotiation.cc:407] Connection header received
0808 04:06:10.617399 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:06:10.617405 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:06:10.617477 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:06:10.617558 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:06:10.619313 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:06:10.620141 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:06:10.622150 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:06:10.622575 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:06:10.622668 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:06:10.625613 (+ 2945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:06:10.625618 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:06:10.625621 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:06:10.625653 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:06:10.628217 (+ 2564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:06:10.628987 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:06:10.628991 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:06:10.628994 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:06:10.629108 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:06:10.629474 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:06:10.629477 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:06:10.629479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:06:10.629726 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:06:10.629970 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0808 04:06:10.630242 (+ 272us) server_negotiation.cc:299] Negotiation successful
0808 04:06:10.630399 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 04:06:10.634948 21767 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51308:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:06:10.641983 21767 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:06:10.684258 22547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:06:10.654510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.3:42495 (local address 127.0.0.1:56682)
0808 04:06:10.672789 (+ 18279us) negotiation.cc:107] Waiting for socket to connect
0808 04:06:10.672806 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:06:10.672937 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:06:10.673654 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:06:10.673663 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:06:10.673699 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:06:10.674173 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:06:10.674192 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:06:10.678817 (+ 4625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:06:10.678823 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:06:10.681177 (+ 2354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:06:10.681194 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:06:10.683353 (+ 2159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:06:10.683359 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:06:10.683563 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:06:10.683617 (+ 54us) client_negotiation.cc:769] Sending connection context
0808 04:06:10.683768 (+ 151us) client_negotiation.cc:240] Negotiation successful
0808 04:06:10.683963 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18161,"thread_start_us":18529,"threads_started":1}
I20260808 04:06:10.685295 22549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:06:10.655110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56682 (local address 127.5.83.3:42495)
0808 04:06:10.672364 (+ 17254us) server_negotiation.cc:206] Beginning negotiation
0808 04:06:10.672372 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:06:10.673049 (+ 677us) server_negotiation.cc:407] Connection header received
0808 04:06:10.673168 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:06:10.673176 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:06:10.673424 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:06:10.673624 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:06:10.674415 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:06:10.678575 (+ 4160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:06:10.681370 (+ 2795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:06:10.682882 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:06:10.683006 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:06:10.683218 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0808 04:06:10.684787 (+ 1569us) server_negotiation.cc:299] Negotiation successful
0808 04:06:10.684950 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17088,"thread_start_us":16925,"threads_started":1}
I20260808 04:06:10.685702 22550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:06:10.654428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:32869 (local address 127.0.0.1:57214)
0808 04:06:10.671440 (+ 17012us) negotiation.cc:107] Waiting for socket to connect
0808 04:06:10.671491 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 04:06:10.671681 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:06:10.674680 (+ 2999us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:06:10.674685 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:06:10.674710 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:06:10.675342 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:06:10.675386 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:06:10.677646 (+ 2260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:06:10.677652 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:06:10.680331 (+ 2679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:06:10.680347 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:06:10.684692 (+ 4345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:06:10.684699 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:06:10.684882 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:06:10.684916 (+ 34us) client_negotiation.cc:769] Sending connection context
0808 04:06:10.685001 (+ 85us) client_negotiation.cc:240] Negotiation successful
0808 04:06:10.685122 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16747,"mutex_wait_us":1,"thread_start_us":28059,"threads_started":1}
I20260808 04:06:10.686573 22548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:06:10.654699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57214 (local address 127.5.83.1:32869)
0808 04:06:10.669967 (+ 15268us) server_negotiation.cc:206] Beginning negotiation
0808 04:06:10.669978 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 04:06:10.673933 (+ 3955us) server_negotiation.cc:407] Connection header received
0808 04:06:10.674041 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:06:10.674048 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:06:10.674235 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:06:10.674424 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:06:10.675673 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:06:10.677477 (+ 1804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:06:10.680525 (+ 3048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:06:10.681987 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:06:10.682106 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:06:10.682335 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0808 04:06:10.686077 (+ 3742us) server_negotiation.cc:299] Negotiation successful
0808 04:06:10.686242 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15044,"thread_start_us":15280,"threads_started":1}
I20260808 04:06:10.689265 22414 tablet_service.cc:1467] Processing CreateTablet for tablet 0bc2ac4f520d4525ab2839dcb49d15ec (DEFAULT_TABLE table=test-table [id=89b68572a6bb4131bf1c85617e2f604e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:06:10.690366 21993 tablet_service.cc:1467] Processing CreateTablet for tablet 0bc2ac4f520d4525ab2839dcb49d15ec (DEFAULT_TABLE table=test-table [id=89b68572a6bb4131bf1c85617e2f604e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:06:10.690923 22414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bc2ac4f520d4525ab2839dcb49d15ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:06:10.691915 22545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:06:10.663657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.2:35155 (local address 127.0.0.1:44590)
0808 04:06:10.682735 (+ 19078us) negotiation.cc:107] Waiting for socket to connect
0808 04:06:10.682752 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:06:10.682971 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:06:10.683818 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:06:10.683823 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:06:10.683848 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:06:10.684178 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:06:10.684190 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:06:10.686163 (+ 1973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:06:10.686168 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:06:10.688136 (+ 1968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:06:10.688153 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:06:10.691173 (+ 3020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:06:10.691179 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:06:10.691339 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:06:10.691370 (+ 31us) client_negotiation.cc:769] Sending connection context
0808 04:06:10.691452 (+ 82us) client_negotiation.cc:240] Negotiation successful
0808 04:06:10.691571 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18962,"thread_start_us":7380,"threads_started":1}
I20260808 04:06:10.691958 21993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bc2ac4f520d4525ab2839dcb49d15ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:06:10.692677 22551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:06:10.664060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44590 (local address 127.5.83.2:35155)
0808 04:06:10.670852 (+ 6792us) server_negotiation.cc:206] Beginning negotiation
0808 04:06:10.670861 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 04:06:10.682867 (+ 12006us) server_negotiation.cc:407] Connection header received
0808 04:06:10.683144 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:06:10.683153 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:06:10.683379 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:06:10.683533 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:06:10.684384 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:06:10.685991 (+ 1607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:06:10.688324 (+ 2333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:06:10.689638 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:06:10.689754 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:06:10.691028 (+ 1274us) server_negotiation.cc:1035] Waiting for connection context
0808 04:06:10.692214 (+ 1186us) server_negotiation.cc:299] Negotiation successful
0808 04:06:10.692377 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6617,"thread_start_us":6499,"threads_started":1}
I20260808 04:06:10.695863 22199 tablet_service.cc:1467] Processing CreateTablet for tablet 0bc2ac4f520d4525ab2839dcb49d15ec (DEFAULT_TABLE table=test-table [id=89b68572a6bb4131bf1c85617e2f604e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:06:10.697185 22199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bc2ac4f520d4525ab2839dcb49d15ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:06:10.720578 22554 tablet_bootstrap.cc:492] T 0bc2ac4f520d4525ab2839dcb49d15ec P 3e0c52ef668d46d3bcabf7a3e17417a2: Bootstrap starting.
I20260808 04:06:10.724531 22555 tablet_bootstrap.cc:492] T 0bc2ac4f520d4525ab2839dcb49d15ec P 847881a547e74ea297246c29704acb99: Bootstrap starting.
I20260808 04:06:10.724619 22554 tablet_bootstrap.cc:654] T 0bc2ac4f520d4525ab2839dcb49d15ec P 3e0c52ef668d46d3bcabf7a3e17417a2: Neither blocks nor log segments found. Creating new log.
I20260808 04:06:10.725925 22554 log.cc:824] T 0bc2ac4f520d4525ab2839dcb49d15ec P 3e0c52ef668d46d3bcabf7a3e17417a2: Log is configured to *not* fsync() on all Append() calls
I20260808 04:06:10.728725 22555 tablet_bootstrap.cc:654] T 0bc2ac4f520d4525ab2839dcb49d15ec P 847881a547e74ea297246c29704acb99: Neither blocks nor log segments found. Creating new log.
I20260808 04:06:10.729653 22557 tablet_bootstrap.cc:492] T 0bc2ac4f520d4525ab2839dcb49d15ec P 4d86eb79e0b34cfc8a4fc7e5f4049f1b: Bootstrap starting.
I20260808 04:06:10.730021 22555 log.cc:824] T 0bc2ac4f520d4525ab2839dcb49d15ec P 847881a547e74ea297246c29704acb99: Log is configured to *not* fsync() on all Append() calls
I20260808 04:06:10.732800 22555 tablet_bootstrap.cc:492] T 0bc2ac4f520d4525ab2839dcb49d15ec P 847881a547e74ea297246c29704acb99: No bootstrap required, opened a new log
I20260808 04:06:10.733120 22555 ts_tablet_manager.cc:1397] T 0bc2ac4f520d4525ab2839dcb49d15ec P 847881a547e74ea297246c29704acb99: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260808 04:06:10.735266 22554 tablet_bootstrap.cc:492] T 0bc2ac4f520d4525ab2839dcb49d15ec P 3e0c52ef668d46d3bcabf7a3e17417a2: No bootstrap required, opened a new log
I20260808 04:06:10.735634 22554 ts_tablet_manager.cc:1397] T 0bc2ac4f520d4525ab2839dcb49d15ec P 3e0c52ef668d46d3bcabf7a3e17417a2: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.005s
I20260808 04:06:10.736707 22557 tablet_bootstrap.cc:654] T 0bc2ac4f520d4525ab2839dcb49d15ec P 4d86eb79e0b34cfc8a4fc7e5f4049f1b: Neither blocks nor log segments found. Creating new log.
I20260808 04:06:10.737828 22557 log.cc:824] T 0bc2ac4f520d4525ab2839dcb49d15ec P 4d86eb79e0b34cfc8a4fc7e5f4049f1b: Log is configured to *not* fsync() on all Append() calls
I20260808 04:06:10.743505 22555 raft_consensus.cc:348] T 0bc2ac4f520d4525ab2839dcb49d15ec P 847881a547e74ea297246c29704acb99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "847881a547e74ea297246c29704acb99" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 32869 } } peers { permanent_uuid: "3e0c52ef668d46d3bcabf7a3e17417a2" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 42495 } } peers { permanent_uuid: "4d86eb79e0b34cfc8a4fc7e5f4049f1b" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 35155 } }
I20260808 04:06:10.744019 22555 raft_consensus.cc:374] T 0bc2ac4f520d4525ab2839dcb49d15ec P 847881a547e74ea297246c29704acb99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:06:10.744127 22555 raft_consensus.cc:729] T 0bc2ac4f520d4525ab2839dcb49d15ec P 847881a547e74ea297246c29704acb99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 847881a547e74ea297246c29704acb99, State: Initialized, Role: FOLLOWER
I20260808 04:06:10.744845 22555 consensus_queue.cc:260] T 0bc2ac4f520d4525ab2839dcb49d15ec P 847881a547e74ea297246c29704acb99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "847881a547e74ea297246c29704acb99" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 32869 } } peers { permanent_uuid: "3e0c52ef668d46d3bcabf7a3e17417a2" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 42495 } } peers { permanent_uuid: "4d86eb79e0b34cfc8a4fc7e5f4049f1b" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 35155 } }
I20260808 04:06:10.748136 22554 raft_consensus.cc:348] T 0bc2ac4f520d4525ab2839dcb49d15ec P 3e0c52ef668d46d3bcabf7a3e17417a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "847881a547e74ea297246c29704acb99" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 32869 } } peers { permanent_uuid: "3e0c52ef668d46d3bcabf7a3e17417a2" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 42495 } } peers { permanent_uuid: "4d86eb79e0b34cfc8a4fc7e5f4049f1b" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 35155 } }
I20260808 04:06:10.748667 22554 raft_consensus.cc:374] T 0bc2ac4f520d4525ab2839dcb49d15ec P 3e0c52ef668d46d3bcabf7a3e17417a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:06:10.749629 22554 raft_consensus.cc:729] T 0bc2ac4f520d4525ab2839dcb49d15ec P 3e0c52ef668d46d3bcabf7a3e17417a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e0c52ef668d46d3bcabf7a3e17417a2, State: Initialized, Role: FOLLOWER
I20260808 04:06:10.751892 22557 tablet_bootstrap.cc:492] T 0bc2ac4f520d4525ab2839dcb49d15ec P 4d86eb79e0b34cfc8a4fc7e5f4049f1b: No bootstrap required, opened a new log
I20260808 04:06:10.752172 22557 ts_tablet_manager.cc:1397] T 0bc2ac4f520d4525ab2839dcb49d15ec P 4d86eb79e0b34cfc8a4fc7e5f4049f1b: Time spent bootstrapping tablet: real 0.023s user 0.010s sys 0.008s
I20260808 04:06:10.752911 22555 ts_tablet_manager.cc:1428] T 0bc2ac4f520d4525ab2839dcb49d15ec P 847881a547e74ea297246c29704acb99: Time spent starting tablet: real 0.020s user 0.009s sys 0.009s
I20260808 04:06:10.753727 22554 consensus_queue.cc:260] T 0bc2ac4f520d4525ab2839dcb49d15ec P 3e0c52ef668d46d3bcabf7a3e17417a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "847881a547e74ea297246c29704acb99" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 32869 } } peers { permanent_uuid: "3e0c52ef668d46d3bcabf7a3e17417a2" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 42495 } } peers { permanent_uuid: "4d86eb79e0b34cfc8a4fc7e5f4049f1b" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 35155 } }
I20260808 04:06:10.760426 22527 heartbeater.cc:503] Master 127.5.83.62:32911 was elected leader, sending a full tablet report...
I20260808 04:06:10.761233 22554 ts_tablet_manager.cc:1428] T 0bc2ac4f520d4525ab2839dcb49d15ec P 3e0c52ef668d46d3bcabf7a3e17417a2: Time spent starting tablet: real 0.025s user 0.010s sys 0.007s
I20260808 04:06:10.762398 22557 raft_consensus.cc:348] T 0bc2ac4f520d4525ab2839dcb49d15ec P 4d86eb79e0b34cfc8a4fc7e5f4049f1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "847881a547e74ea297246c29704acb99" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 32869 } } peers { permanent_uuid: "3e0c52ef668d46d3bcabf7a3e17417a2" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 42495 } } peers { permanent_uuid: "4d86eb79e0b34cfc8a4fc7e5f4049f1b" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 35155 } }
I20260808 04:06:10.762800 22557 raft_consensus.cc:374] T 0bc2ac4f520d4525ab2839dcb49d15ec P 4d86eb79e0b34cfc8a4fc7e5f4049f1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:06:10.762899 22557 raft_consensus.cc:729] T 0bc2ac4f520d4525ab2839dcb49d15ec P 4d86eb79e0b34cfc8a4fc7e5f4049f1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d86eb79e0b34cfc8a4fc7e5f4049f1b, State: Initialized, Role: FOLLOWER
I20260808 04:06:10.763492 22557 consensus_queue.cc:260] T 0bc2ac4f520d4525ab2839dcb49d15ec P 4d86eb79e0b34cfc8a4fc7e5f4049f1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "847881a547e74ea297246c29704acb99" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 32869 } } peers { permanent_uuid: "3e0c52ef668d46d3bcabf7a3e17417a2" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 42495 } } peers { permanent_uuid: "4d86eb79e0b34cfc8a4fc7e5f4049f1b" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 35155 } }
I20260808 04:06:10.764874 22299 heartbeater.cc:503] Master 127.5.83.62:32911 was elected leader, sending a full tablet report...
I20260808 04:06:10.765126 22557 ts_tablet_manager.cc:1428] T 0bc2ac4f520d4525ab2839dcb49d15ec P 4d86eb79e0b34cfc8a4fc7e5f4049f1b: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
W20260808 04:06:10.775247 22304 tablet.cc:2369] T 0bc2ac4f520d4525ab2839dcb49d15ec P 4d86eb79e0b34cfc8a4fc7e5f4049f1b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 04:06:10.829012 22528 tablet.cc:2369] T 0bc2ac4f520d4525ab2839dcb49d15ec P 3e0c52ef668d46d3bcabf7a3e17417a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 04:06:10.976744 22087 tablet.cc:2369] T 0bc2ac4f520d4525ab2839dcb49d15ec P 847881a547e74ea297246c29704acb99: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:06:10.978111 22564 raft_consensus.cc:482] T 0bc2ac4f520d4525ab2839dcb49d15ec P 4d86eb79e0b34cfc8a4fc7e5f4049f1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:06:10.978360 22564 raft_consensus.cc:504] T 0bc2ac4f520d4525ab2839dcb49d15ec P 4d86eb79e0b34cfc8a4fc7e5f4049f1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "847881a547e74ea297246c29704acb99" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 32869 } } peers { permanent_uuid: "3e0c52ef668d46d3bcabf7a3e17417a2" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 42495 } } peers { permanent_uuid: "4d86eb79e0b34cfc8a4fc7e5f4049f1b" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 35155 } }
I20260808 04:06:10.980118 22564 leader_election.cc:288] T 0bc2ac4f520d4525ab2839dcb49d15ec P 4d86eb79e0b34cfc8a4fc7e5f4049f1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 847881a547e74ea297246c29704acb99 (127.5.83.1:32869), 3e0c52ef668d46d3bcabf7a3e17417a2 (127.5.83.3:42495)
I20260808 04:06:10.986994 22549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:06:10.980621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.2:50883 (local address 127.5.83.3:42495)
0808 04:06:10.980837 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0808 04:06:10.980844 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:06:10.981740 (+ 896us) server_negotiation.cc:407] Connection header received
0808 04:06:10.981964 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:06:10.981967 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:06:10.982028 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:06:10.982103 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:06:10.982754 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:06:10.983470 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:06:10.985290 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:06:10.986068 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:06:10.986182 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:06:10.986287 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0808 04:06:10.986618 (+ 331us) server_negotiation.cc:299] Negotiation successful
0808 04:06:10.986751 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 04:06:10.987010 22548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:06:10.980606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.2:36159 (local address 127.5.83.1:32869)
0808 04:06:10.980931 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0808 04:06:10.980936 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:06:10.981699 (+ 763us) server_negotiation.cc:407] Connection header received
0808 04:06:10.981886 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:06:10.981890 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:06:10.981950 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:06:10.982026 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:06:10.982615 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:06:10.983349 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:06:10.985289 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:06:10.986056 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:06:10.986237 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:06:10.986340 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0808 04:06:10.986655 (+ 315us) server_negotiation.cc:299] Negotiation successful
0808 04:06:10.986774 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171}
I20260808 04:06:10.987249 22565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:06:10.980369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:32869 (local address 127.5.83.2:36159)
0808 04:06:10.981578 (+ 1209us) negotiation.cc:107] Waiting for socket to connect
0808 04:06:10.981604 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0808 04:06:10.981750 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:06:10.982057 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:06:10.982061 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:06:10.982085 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:06:10.982464 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:06:10.982474 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:06:10.983480 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:06:10.983485 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:06:10.985142 (+ 1657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:06:10.985154 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:06:10.986352 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:06:10.986359 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:06:10.986514 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:06:10.986537 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 04:06:10.986615 (+ 78us) client_negotiation.cc:240] Negotiation successful
0808 04:06:10.986733 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1068,"mutex_wait_us":47,"spinlock_wait_cycles":30976,"thread_start_us":721,"threads_started":1}
I20260808 04:06:10.987453 22566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:06:10.980591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.3:42495 (local address 127.5.83.2:50883)
0808 04:06:10.981620 (+ 1029us) negotiation.cc:107] Waiting for socket to connect
0808 04:06:10.981635 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:06:10.981822 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:06:10.982127 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:06:10.982132 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:06:10.982158 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:06:10.982605 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:06:10.982614 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:06:10.983612 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:06:10.983616 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:06:10.985147 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:06:10.985159 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:06:10.986293 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:06:10.986300 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:06:10.986458 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:06:10.986499 (+ 41us) client_negotiation.cc:769] Sending connection context
0808 04:06:10.986579 (+ 80us) client_negotiation.cc:240] Negotiation successful
0808 04:06:10.986697 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":929,"mutex_wait_us":33,"spinlock_wait_cycles":23424,"thread_start_us":781,"threads_started":1}
I20260808 04:06:10.988413 22457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bc2ac4f520d4525ab2839dcb49d15ec" candidate_uuid: "4d86eb79e0b34cfc8a4fc7e5f4049f1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e0c52ef668d46d3bcabf7a3e17417a2" is_pre_election: true
I20260808 04:06:10.988436 22021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bc2ac4f520d4525ab2839dcb49d15ec" candidate_uuid: "4d86eb79e0b34cfc8a4fc7e5f4049f1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "847881a547e74ea297246c29704acb99" is_pre_election: true
I20260808 04:06:10.988871 22021 raft_consensus.cc:2454] T 0bc2ac4f520d4525ab2839dcb49d15ec P 847881a547e74ea297246c29704acb99 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d86eb79e0b34cfc8a4fc7e5f4049f1b in term 0.
I20260808 04:06:10.988876 22457 raft_consensus.cc:2454] T 0bc2ac4f520d4525ab2839dcb49d15ec P 3e0c52ef668d46d3bcabf7a3e17417a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d86eb79e0b34cfc8a4fc7e5f4049f1b in term 0.
I20260808 04:06:10.989465 22137 leader_election.cc:302] T 0bc2ac4f520d4525ab2839dcb49d15ec P 4d86eb79e0b34cfc8a4fc7e5f4049f1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d86eb79e0b34cfc8a4fc7e5f4049f1b, 847881a547e74ea297246c29704acb99; no voters:
I20260808 04:06:10.989796 22564 raft_consensus.cc:2781] T 0bc2ac4f520d4525ab2839dcb49d15ec P 4d86eb79e0b34cfc8a4fc7e5f4049f1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:06:10.989926 22564 raft_consensus.cc:482] T 0bc2ac4f520d4525ab2839dcb49d15ec P 4d86eb79e0b34cfc8a4fc7e5f4049f1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:06:10.990022 22564 raft_consensus.cc:3037] T 0bc2ac4f520d4525ab2839dcb49d15ec P 4d86eb79e0b34cfc8a4fc7e5f4049f1b [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:06:10.992305 22564 raft_consensus.cc:504] T 0bc2ac4f520d4525ab2839dcb49d15ec P 4d86eb79e0b34cfc8a4fc7e5f4049f1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "847881a547e74ea297246c29704acb99" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 32869 } } peers { permanent_uuid: "3e0c52ef668d46d3bcabf7a3e17417a2" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 42495 } } peers { permanent_uuid: "4d86eb79e0b34cfc8a4fc7e5f4049f1b" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 35155 } }
I20260808 04:06:10.992871 22564 leader_election.cc:288] T 0bc2ac4f520d4525ab2839dcb49d15ec P 4d86eb79e0b34cfc8a4fc7e5f4049f1b [CANDIDATE]: Term 1 election: Requested vote from peers 847881a547e74ea297246c29704acb99 (127.5.83.1:32869), 3e0c52ef668d46d3bcabf7a3e17417a2 (127.5.83.3:42495)
I20260808 04:06:10.993243 22021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bc2ac4f520d4525ab2839dcb49d15ec" candidate_uuid: "4d86eb79e0b34cfc8a4fc7e5f4049f1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "847881a547e74ea297246c29704acb99"
I20260808 04:06:10.993325 22457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bc2ac4f520d4525ab2839dcb49d15ec" candidate_uuid: "4d86eb79e0b34cfc8a4fc7e5f4049f1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e0c52ef668d46d3bcabf7a3e17417a2"
I20260808 04:06:10.993418 22021 raft_consensus.cc:3037] T 0bc2ac4f520d4525ab2839dcb49d15ec P 847881a547e74ea297246c29704acb99 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:06:10.993515 22457 raft_consensus.cc:3037] T 0bc2ac4f520d4525ab2839dcb49d15ec P 3e0c52ef668d46d3bcabf7a3e17417a2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:06:10.995683 22021 raft_consensus.cc:2454] T 0bc2ac4f520d4525ab2839dcb49d15ec P 847881a547e74ea297246c29704acb99 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d86eb79e0b34cfc8a4fc7e5f4049f1b in term 1.
I20260808 04:06:10.995687 22457 raft_consensus.cc:2454] T 0bc2ac4f520d4525ab2839dcb49d15ec P 3e0c52ef668d46d3bcabf7a3e17417a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d86eb79e0b34cfc8a4fc7e5f4049f1b in term 1.
I20260808 04:06:10.996102 22137 leader_election.cc:302] T 0bc2ac4f520d4525ab2839dcb49d15ec P 4d86eb79e0b34cfc8a4fc7e5f4049f1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d86eb79e0b34cfc8a4fc7e5f4049f1b, 847881a547e74ea297246c29704acb99; no voters:
I20260808 04:06:10.996344 22564 raft_consensus.cc:2781] T 0bc2ac4f520d4525ab2839dcb49d15ec P 4d86eb79e0b34cfc8a4fc7e5f4049f1b [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:06:10.996732 22564 raft_consensus.cc:686] T 0bc2ac4f520d4525ab2839dcb49d15ec P 4d86eb79e0b34cfc8a4fc7e5f4049f1b [term 1 LEADER]: Becoming Leader. State: Replica: 4d86eb79e0b34cfc8a4fc7e5f4049f1b, State: Running, Role: LEADER
I20260808 04:06:10.997226 22564 consensus_queue.cc:237] T 0bc2ac4f520d4525ab2839dcb49d15ec P 4d86eb79e0b34cfc8a4fc7e5f4049f1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "847881a547e74ea297246c29704acb99" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 32869 } } peers { permanent_uuid: "3e0c52ef668d46d3bcabf7a3e17417a2" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 42495 } } peers { permanent_uuid: "4d86eb79e0b34cfc8a4fc7e5f4049f1b" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 35155 } }
I20260808 04:06:11.001806 21767 catalog_manager.cc:5495] T 0bc2ac4f520d4525ab2839dcb49d15ec P 4d86eb79e0b34cfc8a4fc7e5f4049f1b reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d86eb79e0b34cfc8a4fc7e5f4049f1b (127.5.83.2). New cstate: current_term: 1 leader_uuid: "4d86eb79e0b34cfc8a4fc7e5f4049f1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "847881a547e74ea297246c29704acb99" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 32869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e0c52ef668d46d3bcabf7a3e17417a2" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 42495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d86eb79e0b34cfc8a4fc7e5f4049f1b" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 35155 } health_report { overall_health: HEALTHY } } }
I20260808 04:06:11.037667 22551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:06:11.032447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44606 (local address 127.5.83.2:35155)
0808 04:06:11.032649 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0808 04:06:11.032656 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:06:11.032854 (+ 198us) server_negotiation.cc:407] Connection header received
0808 04:06:11.033013 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:06:11.033017 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:06:11.033151 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:06:11.033279 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:06:11.033616 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:06:11.034360 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:06:11.035744 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:06:11.036131 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:06:11.036221 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:06:11.036678 (+ 457us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:06:11.037160 (+ 482us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:06:11.037269 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0808 04:06:11.037346 (+ 77us) server_negotiation.cc:299] Negotiation successful
0808 04:06:11.037410 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 04:06:11.045724 22457 raft_consensus.cc:1261] T 0bc2ac4f520d4525ab2839dcb49d15ec P 3e0c52ef668d46d3bcabf7a3e17417a2 [term 1 FOLLOWER]: Refusing update from remote peer 4d86eb79e0b34cfc8a4fc7e5f4049f1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:06:11.045766 22021 raft_consensus.cc:1261] T 0bc2ac4f520d4525ab2839dcb49d15ec P 847881a547e74ea297246c29704acb99 [term 1 FOLLOWER]: Refusing update from remote peer 4d86eb79e0b34cfc8a4fc7e5f4049f1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:06:11.046733 22568 consensus_queue.cc:1035] T 0bc2ac4f520d4525ab2839dcb49d15ec P 4d86eb79e0b34cfc8a4fc7e5f4049f1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e0c52ef668d46d3bcabf7a3e17417a2" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 42495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:06:11.047065 22564 consensus_queue.cc:1035] T 0bc2ac4f520d4525ab2839dcb49d15ec P 4d86eb79e0b34cfc8a4fc7e5f4049f1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "847881a547e74ea297246c29704acb99" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 32869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:06:11.059137 22572 mvcc.cc:205] Tried to move back new op lower bound from 7316119433392955392 to 7316119433208352768. Current Snapshot: MvccSnapshot[applied={T|T < 7316119433392955392}]
I20260808 04:06:11.072803 21767 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:06:11.073149 21767 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:06:11.076843 21767 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cf6929c542314cd3a2bf112a7ce508d2: Sending DeleteTablet for 3 replicas of tablet 0bc2ac4f520d4525ab2839dcb49d15ec
I20260808 04:06:11.077862 21993 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bc2ac4f520d4525ab2839dcb49d15ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:06:11 UTC) from {username='kudu', principal='kudu/127.5.83.62@KRBTEST.COM'} at 127.0.0.1:57214
I20260808 04:06:11.077893 22414 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bc2ac4f520d4525ab2839dcb49d15ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:06:11 UTC) from {username='kudu', principal='kudu/127.5.83.62@KRBTEST.COM'} at 127.0.0.1:56682
I20260808 04:06:11.078187 22199 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bc2ac4f520d4525ab2839dcb49d15ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:06:11 UTC) from {username='kudu', principal='kudu/127.5.83.62@KRBTEST.COM'} at 127.0.0.1:44590
I20260808 04:06:11.081385 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 21890
I20260808 04:06:11.084192 22588 tablet_replica.cc:330] T 0bc2ac4f520d4525ab2839dcb49d15ec P 3e0c52ef668d46d3bcabf7a3e17417a2: stopping tablet replica
I20260808 04:06:11.084362 22586 tablet_replica.cc:330] T 0bc2ac4f520d4525ab2839dcb49d15ec P 847881a547e74ea297246c29704acb99: stopping tablet replica
I20260808 04:06:11.084627 22588 raft_consensus.cc:2229] T 0bc2ac4f520d4525ab2839dcb49d15ec P 3e0c52ef668d46d3bcabf7a3e17417a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:06:11.084710 22586 raft_consensus.cc:2229] T 0bc2ac4f520d4525ab2839dcb49d15ec P 847881a547e74ea297246c29704acb99 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:06:11.085132 22588 raft_consensus.cc:2258] T 0bc2ac4f520d4525ab2839dcb49d15ec P 3e0c52ef668d46d3bcabf7a3e17417a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:06:11.085132 22586 raft_consensus.cc:2258] T 0bc2ac4f520d4525ab2839dcb49d15ec P 847881a547e74ea297246c29704acb99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:06:11.087447 22586 ts_tablet_manager.cc:1905] T 0bc2ac4f520d4525ab2839dcb49d15ec P 847881a547e74ea297246c29704acb99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:06:11.093391 22588 ts_tablet_manager.cc:1905] T 0bc2ac4f520d4525ab2839dcb49d15ec P 3e0c52ef668d46d3bcabf7a3e17417a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:06:11.094434 22586 ts_tablet_manager.cc:1918] T 0bc2ac4f520d4525ab2839dcb49d15ec P 847881a547e74ea297246c29704acb99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:06:11.094599 22586 log.cc:1196] T 0bc2ac4f520d4525ab2839dcb49d15ec P 847881a547e74ea297246c29704acb99: Deleting WAL directory at /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/ts-0/wal/wals/0bc2ac4f520d4525ab2839dcb49d15ec
I20260808 04:06:11.095069 22586 ts_tablet_manager.cc:1939] T 0bc2ac4f520d4525ab2839dcb49d15ec P 847881a547e74ea297246c29704acb99: Deleting consensus metadata
I20260808 04:06:11.096985 21747 catalog_manager.cc:4841] TS 847881a547e74ea297246c29704acb99 (127.5.83.1:32869): tablet 0bc2ac4f520d4525ab2839dcb49d15ec (table test-table [id=89b68572a6bb4131bf1c85617e2f604e]) successfully deleted
I20260808 04:06:11.098006 22589 tablet_replica.cc:330] T 0bc2ac4f520d4525ab2839dcb49d15ec P 4d86eb79e0b34cfc8a4fc7e5f4049f1b: stopping tablet replica
I20260808 04:06:11.098435 22589 raft_consensus.cc:2229] T 0bc2ac4f520d4525ab2839dcb49d15ec P 4d86eb79e0b34cfc8a4fc7e5f4049f1b [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:06:11.099109 22589 raft_consensus.cc:2258] T 0bc2ac4f520d4525ab2839dcb49d15ec P 4d86eb79e0b34cfc8a4fc7e5f4049f1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:06:11.099786 22588 ts_tablet_manager.cc:1918] T 0bc2ac4f520d4525ab2839dcb49d15ec P 3e0c52ef668d46d3bcabf7a3e17417a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:06:11.099953 22588 log.cc:1196] T 0bc2ac4f520d4525ab2839dcb49d15ec P 3e0c52ef668d46d3bcabf7a3e17417a2: Deleting WAL directory at /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/ts-2/wal/wals/0bc2ac4f520d4525ab2839dcb49d15ec
I20260808 04:06:11.100394 22588 ts_tablet_manager.cc:1939] T 0bc2ac4f520d4525ab2839dcb49d15ec P 3e0c52ef668d46d3bcabf7a3e17417a2: Deleting consensus metadata
I20260808 04:06:11.101279 22589 ts_tablet_manager.cc:1905] T 0bc2ac4f520d4525ab2839dcb49d15ec P 4d86eb79e0b34cfc8a4fc7e5f4049f1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:06:11.102082 21751 catalog_manager.cc:4841] TS 3e0c52ef668d46d3bcabf7a3e17417a2 (127.5.83.3:42495): tablet 0bc2ac4f520d4525ab2839dcb49d15ec (table test-table [id=89b68572a6bb4131bf1c85617e2f604e]) successfully deleted
Aug 08 04:06:11 dist-test-slave-1l3l krb5kdc[21036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.1@KRBTEST.COM
I20260808 04:06:11.108387 22589 ts_tablet_manager.cc:1918] T 0bc2ac4f520d4525ab2839dcb49d15ec P 4d86eb79e0b34cfc8a4fc7e5f4049f1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:06:11.108551 22589 log.cc:1196] T 0bc2ac4f520d4525ab2839dcb49d15ec P 4d86eb79e0b34cfc8a4fc7e5f4049f1b: Deleting WAL directory at /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161826605655-5452-0/minicluster-data/ts-1/wal/wals/0bc2ac4f520d4525ab2839dcb49d15ec
I20260808 04:06:11.109119 22589 ts_tablet_manager.cc:1939] T 0bc2ac4f520d4525ab2839dcb49d15ec P 4d86eb79e0b34cfc8a4fc7e5f4049f1b: Deleting consensus metadata
I20260808 04:06:11.110806 21749 catalog_manager.cc:4841] TS 4d86eb79e0b34cfc8a4fc7e5f4049f1b (127.5.83.2:35155): tablet 0bc2ac4f520d4525ab2839dcb49d15ec (table test-table [id=89b68572a6bb4131bf1c85617e2f604e]) successfully deleted
I20260808 04:06:11.111594 22548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:06:11.083357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57230 (local address 127.5.83.1:32869)
0808 04:06:11.083544 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0808 04:06:11.083550 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:06:11.094074 (+ 10524us) server_negotiation.cc:407] Connection header received
0808 04:06:11.094350 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:06:11.094357 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:06:11.094462 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:06:11.094541 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:06:11.097560 (+ 3019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:06:11.098686 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:06:11.100880 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:06:11.101481 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:06:11.101580 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:06:11.105880 (+ 4300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:06:11.105893 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:06:11.105907 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:06:11.105937 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:06:11.108840 (+ 2903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:06:11.109526 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:06:11.109533 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:06:11.109535 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:06:11.109658 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:06:11.110049 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:06:11.110053 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:06:11.110055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:06:11.110408 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:06:11.110684 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0808 04:06:11.111056 (+ 372us) server_negotiation.cc:299] Negotiation successful
0808 04:06:11.111284 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 04:06:11.112236 22079 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:06:11.200932 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 21890
I20260808 04:06:11.221462 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 22093
Aug 08 04:06:11 dist-test-slave-1l3l krb5kdc[21036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.2@KRBTEST.COM
I20260808 04:06:11.237902 22551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:06:11.222234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44616 (local address 127.5.83.2:35155)
0808 04:06:11.222419 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0808 04:06:11.222425 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:06:11.222586 (+ 161us) server_negotiation.cc:407] Connection header received
0808 04:06:11.222770 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:06:11.222776 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:06:11.222847 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:06:11.222967 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:06:11.224365 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:06:11.225209 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:06:11.226556 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:06:11.227023 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:06:11.227148 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:06:11.231666 (+ 4518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:06:11.231679 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:06:11.231694 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:06:11.231755 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:06:11.234881 (+ 3126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:06:11.235831 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:06:11.235838 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:06:11.235841 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:06:11.236022 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:06:11.236422 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:06:11.236425 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:06:11.236427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:06:11.236765 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:06:11.237026 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0808 04:06:11.237390 (+ 364us) server_negotiation.cc:299] Negotiation successful
0808 04:06:11.237561 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260808 04:06:11.238647 22293 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:06:11.349509 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 22093
I20260808 04:06:11.369915 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 22310
Aug 08 04:06:11 dist-test-slave-1l3l krb5kdc[21036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.3@KRBTEST.COM
I20260808 04:06:11.384919 22549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:06:11.370732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56698 (local address 127.5.83.3:42495)
0808 04:06:11.370929 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0808 04:06:11.370934 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:06:11.370954 (+ 20us) server_negotiation.cc:407] Connection header received
0808 04:06:11.371124 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:06:11.371128 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:06:11.371193 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:06:11.371277 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:06:11.372661 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:06:11.373377 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:06:11.374726 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:06:11.375109 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:06:11.375218 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:06:11.379506 (+ 4288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:06:11.379521 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:06:11.379536 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:06:11.379576 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:06:11.382168 (+ 2592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:06:11.382793 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:06:11.382801 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:06:11.382803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:06:11.382961 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:06:11.383326 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:06:11.383330 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:06:11.383332 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:06:11.383756 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:06:11.384067 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0808 04:06:11.384376 (+ 309us) server_negotiation.cc:299] Negotiation successful
0808 04:06:11.384582 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 04:06:11.385510 22521 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:06:11.480293 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 22310
I20260808 04:06:11.501232 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 21724
I20260808 04:06:11.502427 21808 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:06:11.612567 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 21724
I20260808 04:06:11.649796 5452 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260808 04:06:11.994190 5452 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260808 04:06:11.994465 5452 mini_ranger.cc:66] Stopping Ranger...
I20260808 04:06:12.338467 5452 mini_ranger.cc:68] Stopped Ranger
2026-08-08 04:06:12.338 UTC [21081] LOG: received smart shutdown request
2026-08-08 04:06:12.342 UTC [21081] LOG: background worker "logical replication launcher" (PID 21093) exited with exit code 1
2026-08-08 04:06:12.342 UTC [21088] LOG: shutting down
2026-08-08 04:06:12.480 UTC [21081] LOG: database system is shut down
2026-08-08T04:06:12Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (47721 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:06:12 dist-test-slave-1l3l krb5kdc[22611](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:06:12 dist-test-slave-1l3l krb5kdc[22611](info): set up 2 sockets
Aug 08 04:06:12 dist-test-slave-1l3l krb5kdc[22611](info): commencing operation
krb5kdc: starting...
W20260808 04:06:13.784809 5452 mini_kdc.cc:121] Time spent starting KDC: real 1.152s user 0.005s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/krb5kdc/test-user.keytab.
Aug 08 04:06:13 dist-test-slave-1l3l krb5kdc[22611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161973, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:06:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:06:13Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.5.83.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.5.83.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.5.83.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/krb5kdc/rangeradmin_127.5.83.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.5.83.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/krb5kdc/rangeradmin_127.5.83.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.5.83.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.5.83.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.5.83.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/krb5kdc/rangerlookup_127.5.83.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.5.83.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/krb5kdc/rangerlookup_127.5.83.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.5.83.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.5.83.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.83.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/krb5kdc/HTTP_127.5.83.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.5.83.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/krb5kdc/HTTP_127.5.83.20@KRBTEST.COM.keytab.
I20260808 04:06:14.096493 5452 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task2cd2tT/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/postgres -l logfile start
2026-08-08 04:06:16.379 UTC [22662] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-08 04:06:16.379 UTC [22662] LOG: listening on IPv4 address "127.5.83.20", port 55667
2026-08-08 04:06:16.382 UTC [22662] LOG: listening on Unix socket "/tmp/.s.PGSQL.55667"
2026-08-08 04:06:16.403 UTC [22666] LOG: database system was shut down at 2026-08-08 04:06:15 UTC
2026-08-08 04:06:16.407 UTC [22662] LOG: database system is ready to accept connections
I20260808 04:06:17.469496 5452 mini_postgres.cc:96] Postgres bound to 55667
2026-08-08 04:06:17.489 UTC [22676] FATAL: database "slave" does not exist
127.5.83.20:55667 - accepting connections
I20260808 04:06:17.490513 5452 mini_ranger.cc:161] Starting Ranger...
I20260808 04:06:17.516551 5452 mini_ranger.cc:84] Created miniranger Postgres user
I20260808 04:06:17.929061 5452 mini_ranger.cc:87] Created ranger Postgres database
I20260808 04:06:17.929184 5452 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ranger-admin
2026-08-08 04:06:18,207 [I] DB FLAVOR :POSTGRES
2026-08-08 04:06:18,208 [I] --------- Verifying Ranger DB connection ---------
2026-08-08 04:06:18,208 [I] Checking connection..
2026-08-08 04:06:18,208 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2cd2tT/build/asan/bin/postgresql.jar:/tmp/dist-test-task2cd2tT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.20:55667/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-08 04:06:18,455 [I] Checking connection passed.
2026-08-08 04:06:18,456 [I] --------- Verifying version history table ---------
2026-08-08 04:06:18,456 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2cd2tT/build/asan/bin/postgresql.jar:/tmp/dist-test-task2cd2tT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.20:55667/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 04:06:18,692 [I] Table x_db_version_h does not exist in database ranger
2026-08-08 04:06:18,692 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2cd2tT/build/asan/bin/postgresql.jar:/tmp/dist-test-task2cd2tT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.20:55667/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 04:06:18,928 [I] Table x_db_version_h does not exist in database ranger
2026-08-08 04:06:18,928 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-08 04:06:18,928 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2cd2tT/build/asan/bin/postgresql.jar:/tmp/dist-test-task2cd2tT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.20:55667/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task2cd2tT/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-08 04:06:19.166 UTC [22771] WARNING: there is no transaction in progress
2026-08-08 04:06:19,174 [I] create_dbversion_catalog.sql file imported successfully
2026-08-08 04:06:19,175 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2cd2tT/build/asan/bin/postgresql.jar:/tmp/dist-test-task2cd2tT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.20:55667/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 04:06:19,429 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-08 04:06:19,429 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-08 04:06:19,429 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2cd2tT/build/asan/bin/postgresql.jar:/tmp/dist-test-task2cd2tT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.20:55667/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-08 04:06:19,688 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2cd2tT/build/asan/bin/postgresql.jar:/tmp/dist-test-task2cd2tT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.20:55667/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-08 04:06:19,940 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2cd2tT/build/asan/bin/postgresql.jar:/tmp/dist-test-task2cd2tT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.20:55667/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-1l3l.c.gcp-upstream.internal','N') ;"
2026-08-08 04:06:20,181 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2cd2tT/build/asan/bin/postgresql.jar:/tmp/dist-test-task2cd2tT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.20:55667/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-08 04:06:20,427 [I] Table x_portal_user does not exist in database ranger
2026-08-08 04:06:20,427 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2cd2tT/build/asan/bin/postgresql.jar:/tmp/dist-test-task2cd2tT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.20:55667/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-08 04:06:20,671 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-08 04:06:20,671 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2cd2tT/build/asan/bin/postgresql.jar:/tmp/dist-test-task2cd2tT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.20:55667/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-08 04:06:20,906 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2cd2tT/build/asan/bin/postgresql.jar:/tmp/dist-test-task2cd2tT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.20:55667/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-08 04:06:21,144 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-08 04:06:21,145 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2cd2tT/build/asan/bin/postgresql.jar:/tmp/dist-test-task2cd2tT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.20:55667/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task2cd2tT/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-08 04:06:22.033 UTC [22943] WARNING: there is no transaction in progress
2026-08-08 04:06:22.051 UTC [22943] WARNING: there is no transaction in progress
2026-08-08 04:06:22.066 UTC [22943] WARNING: there is no transaction in progress
2026-08-08 04:06:22.080 UTC [22943] WARNING: there is no transaction in progress
2026-08-08 04:06:22.097 UTC [22943] WARNING: there is no transaction in progress
2026-08-08 04:06:22.115 UTC [22943] WARNING: there is no transaction in progress
2026-08-08 04:06:22.176 UTC [22943] WARNING: there is no transaction in progress
2026-08-08 04:06:22.182 UTC [22943] WARNING: there is no transaction in progress
2026-08-08 04:06:22.195 UTC [22943] WARNING: there is no transaction in progress
2026-08-08 04:06:22.204 UTC [22943] WARNING: there is no transaction in progress
2026-08-08 04:06:22.213 UTC [22943] WARNING: there is no transaction in progress
2026-08-08 04:06:22.225 UTC [22943] WARNING: there is no transaction in progress
2026-08-08 04:06:22.234 UTC [22943] WARNING: there is no transaction in progress
2026-08-08 04:06:22.240 UTC [22943] WARNING: there is no transaction in progress
2026-08-08 04:06:22.246 UTC [22943] WARNING: there is no transaction in progress
2026-08-08 04:06:22.252 UTC [22943] WARNING: there is no transaction in progress
2026-08-08 04:06:22,787 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-08 04:06:22,787 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2cd2tT/build/asan/bin/postgresql.jar:/tmp/dist-test-task2cd2tT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.20:55667/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-08 04:06:23,024 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-08 04:06:23,025 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2cd2tT/build/asan/bin/postgresql.jar:/tmp/dist-test-task2cd2tT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.20:55667/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-08 04:06:23,281 [I] Table x_portal_user already exists in database 'ranger'
2026-08-08 04:06:23,281 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2cd2tT/build/asan/bin/postgresql.jar:/tmp/dist-test-task2cd2tT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.20:55667/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-08 04:06:23,536 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-08 04:06:23,537 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2cd2tT/build/asan/bin/postgresql.jar:/tmp/dist-test-task2cd2tT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.20:55667/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-08 04:06:23,790 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2cd2tT/build/asan/bin/postgresql.jar:/tmp/dist-test-task2cd2tT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.20:55667/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-08 04:06:24,045 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2cd2tT/build/asan/bin/postgresql.jar:/tmp/dist-test-task2cd2tT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.20:55667/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-1l3l.c.gcp-upstream.internal';"
2026-08-08 04:06:24,293 [I] CORE_DB_SCHEMA import status has been updated
2026-08-08 04:06:24,294 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2cd2tT/build/asan/bin/postgresql.jar:/tmp/dist-test-task2cd2tT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.20:55667/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-08 04:06:24,534 [I] DB_PATCHES have already been applied
I20260808 04:06:24.540237 5452 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ranger-admin:/tmp/dist-test-task2cd2tT/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task2cd2tT/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task2cd2tT/build/asan/bin/postgresql.jar:/tmp/dist-test-task2cd2tT/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260808 04:06:24.540373 5452 mini_ranger.cc:193] Using host: 127.5.83.20
I20260808 04:06:24.558856 5452 mini_ranger.cc:227] Ranger admin URL: http://127.5.83.20:45593
Aug 08, 2026 4:06:24 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 08, 2026 4:06:24 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 08, 2026 4:06:24 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ranger-admin/ews/webapp
Aug 08, 2026 4:06:24 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 08, 2026 4:06:24 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 08, 2026 4:06:25 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ranger-admin/ews/webapp].
Aug 08, 2026 4:06:25 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260808 04:06:25.774366 5452 mini_ranger.cc:160] Time spent starting Ranger: real 8.284s user 0.001s sys 0.047s
Aug 08, 2026 4:06:45 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 08, 2026 4:06:45 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 08, 2026 4:06:45 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 08, 2026 4:06:45 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 08, 2026 4:06:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 08, 2026 4:06:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 08, 2026 4:06:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 08, 2026 4:06:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 08, 2026 4:06:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 08, 2026 4:06:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 08, 2026 4:06:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 08, 2026 4:06:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 08, 2026 4:06:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 08, 2026 4:06:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 08, 2026 4:06:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 08, 2026 4:06:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 08, 2026 4:06:45 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 08, 2026 4:06:46 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260808 04:06:46.681176 5452 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.5.83.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.5.83.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.5.83.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/krb5kdc/rangerkms_127.5.83.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.5.83.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/krb5kdc/rangerkms_127.5.83.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.5.83.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.5.83.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.83.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/krb5kdc/HTTP_127.5.83.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.5.83.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/krb5kdc/HTTP_127.5.83.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 08 04:06:46 dist-test-slave-1l3l krb5kdc[22611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162006, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260808 04:06:46.883066 5452 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260808 04:06:46.907404 5452 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260808 04:06:47.644536 5452 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260808 04:06:47.644644 5452 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ranger-kms
I20260808 04:06:47.644685 5452 mini_ranger_kms.cc:227] Using postgres at 127.5.83.20:55667
2026-08-08 04:06:48,003 [I] DB FLAVOR :POSTGRES
2026-08-08 04:06:48,004 [I] --------- Verifying Ranger DB connection ---------
2026-08-08 04:06:48,004 [I] Checking connection
2026-08-08 04:06:48,004 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2cd2tT/build/asan/bin/postgresql.jar:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.20:55667/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-08 04:06:48,233 [I] connection success
2026-08-08 04:06:48,233 [I] --------- Verifying Ranger DB tables ---------
2026-08-08 04:06:48,233 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-08 04:06:48,233 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2cd2tT/build/asan/bin/postgresql.jar:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.20:55667/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-08 04:06:48,474 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-08 04:06:48,474 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-08 04:06:48,474 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-08 04:06:48,474 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2cd2tT/build/asan/bin/postgresql.jar:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.20:55667/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-08 04:06:48,716 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260808 04:06:49.136523 5452 mini_ranger_kms.cc:310] Created kms service
I20260808 04:06:49.325026 5452 mini_ranger_kms.cc:326] Created kudu user
I20260808 04:06:49.364127 5452 mini_ranger_kms.cc:343] Created rangerkms user
Aug 08 04:06:49 dist-test-slave-1l3l krb5kdc[22611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162006, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.5.83.20@KRBTEST.COM
I20260808 04:06:49.754920 5452 mini_ranger_kms.cc:384] Added ranger policy
I20260808 04:06:49.755230 5452 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ranger-kms:/tmp/dist-test-task2cd2tT/build/asan/bin/postgresql.jar:/tmp/dist-test-task2cd2tT/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task2cd2tT/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260808 04:06:49.755286 5452 mini_ranger_kms.cc:240] Using host: 127.5.83.20
I20260808 04:06:49.774101 5452 mini_ranger_kms.cc:276] Ranger KMS PID: 23215
I20260808 04:06:49.774228 5452 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.5.83.20:59983
Aug 08, 2026 4:06:50 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 08, 2026 4:06:50 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 08, 2026 4:06:51 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 08, 2026 4:06:51 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 08, 2026 4:06:51 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ranger-kms/ews/webapp].
Aug 08, 2026 4:06:51 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 08 04:06:52 dist-test-slave-1l3l krb5kdc[22611](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786162012, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.83.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:06:52.322803 5452 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.440s user 0.080s sys 0.306s
I20260808 04:06:52.323077 5452 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260808 04:06:52.323266 5452 mini_ranger_kms.cc:401] 127.5.83.20:59983/kms/v1/keys
Aug 08 04:06:52 dist-test-slave-1l3l krb5kdc[22611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162012, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.83.20@KRBTEST.COM for HTTP/127.5.83.20@KRBTEST.COM
Aug 08 04:06:52 dist-test-slave-1l3l krb5kdc[22611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162012, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.83.20@KRBTEST.COM for HTTP/127.5.83.20@KRBTEST.COM
Aug 08 04:06:54 dist-test-slave-1l3l krb5kdc[22611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.5.83.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab.
Entry for principal kudu/127.5.83.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab.
I20260808 04:06:55.058671 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:43283
--webserver_interface=127.5.83.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab
--principal=kudu/127.5.83.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:35565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.62:43283
--ranger_config_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.83.20:59983/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:06:55.393682 23287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:06:55.393944 23287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:06:55.394081 23287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:06:55.401129 23287 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:06:55.401224 23287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:06:55.401297 23287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:06:55.401372 23287 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:06:55.401444 23287 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:06:55.411113 23287 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:35565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/master-0/wal
--ranger_kms_url=127.5.83.20:59983/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.5.83.62:43283
--ranger_config_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab
--principal=kudu/127.5.83.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:43283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:06:55.412281 23287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:06:55.413524 23287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:06:55.424582 23295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:06:55.424542 23300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:06:55.424616 23293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:06:55.425508 23287 server_base.cc:1034] running on GCE node
I20260808 04:06:55.426208 23287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:06:55.427488 23287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:06:55.428680 23287 hybrid_clock.cc:648] HybridClock initialized: now 1786162015428639 us; error 32 us; skew 500 ppm
I20260808 04:06:55.428997 23287 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:06:55 dist-test-slave-1l3l krb5kdc[22611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162015, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:06:55.435611 23287 init.cc:377] Logged in from keytab as kudu/127.5.83.62@KRBTEST.COM (short username kudu)
I20260808 04:06:55.437022 23287 webserver.cc:460] Webserver started at http://127.5.83.62:46183/ using document root <none> and password file <none>
I20260808 04:06:55.437628 23287 fs_manager.cc:359] Metadata directory not provided
I20260808 04:06:55.437726 23287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:06:55.437978 23287 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 04:06:55 dist-test-slave-1l3l krb5kdc[22611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162015, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.62@KRBTEST.COM for HTTP/127.5.83.20@KRBTEST.COM
I20260808 04:06:55.501528 23287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/master-0/data/instance:
uuid: "4ce7014684154846afcdf2e7a618d8b5"
format_stamp: "Formatted at 2026-08-08 04:06:55 on dist-test-slave-1l3l"
server_key: "4131bc8af29cb25a8d38e2d5fc20cb69"
server_key_iv: "268bb724d13f17d31028f72cf1f2eee9"
server_key_version: "kuduclusterkey@0"
I20260808 04:06:55.502353 23287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/master-0/wal/instance:
uuid: "4ce7014684154846afcdf2e7a618d8b5"
format_stamp: "Formatted at 2026-08-08 04:06:55 on dist-test-slave-1l3l"
server_key: "4131bc8af29cb25a8d38e2d5fc20cb69"
server_key_iv: "268bb724d13f17d31028f72cf1f2eee9"
server_key_version: "kuduclusterkey@0"
I20260808 04:06:55.507619 23287 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260808 04:06:55.526038 23310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:06:55.527387 23287 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260808 04:06:55.527562 23287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/master-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/master-0/wal
uuid: "4ce7014684154846afcdf2e7a618d8b5"
format_stamp: "Formatted at 2026-08-08 04:06:55 on dist-test-slave-1l3l"
server_key: "4131bc8af29cb25a8d38e2d5fc20cb69"
server_key_iv: "268bb724d13f17d31028f72cf1f2eee9"
server_key_version: "kuduclusterkey@0"
I20260808 04:06:55.527803 23287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:06:55.563684 23287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:06:55.567624 23287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:06:55.567941 23287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:06:55.586733 23287 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.62:43283
I20260808 04:06:55.586753 23392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.62:43283 every 8 connection(s)
I20260808 04:06:55.588097 23287 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
I20260808 04:06:55.591986 23393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:06:55.594118 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 23287
I20260808 04:06:55.594341 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/master-0/wal/instance
I20260808 04:06:55.595284 5452 external_mini_cluster.cc:1428] Setting key 6b1b96a0d8b69870a712c8ffd60ae143
I20260808 04:06:55.605756 23393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ce7014684154846afcdf2e7a618d8b5: Bootstrap starting.
I20260808 04:06:55.609969 23393 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ce7014684154846afcdf2e7a618d8b5: Neither blocks nor log segments found. Creating new log.
I20260808 04:06:55.611549 23393 log.cc:824] T 00000000000000000000000000000000 P 4ce7014684154846afcdf2e7a618d8b5: Log is configured to *not* fsync() on all Append() calls
I20260808 04:06:55.614746 23393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ce7014684154846afcdf2e7a618d8b5: No bootstrap required, opened a new log
I20260808 04:06:55.624130 23393 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ce7014684154846afcdf2e7a618d8b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ce7014684154846afcdf2e7a618d8b5" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 43283 } }
I20260808 04:06:55.624584 23393 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ce7014684154846afcdf2e7a618d8b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:06:55.624714 23393 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ce7014684154846afcdf2e7a618d8b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ce7014684154846afcdf2e7a618d8b5, State: Initialized, Role: FOLLOWER
I20260808 04:06:55.625434 23393 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ce7014684154846afcdf2e7a618d8b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ce7014684154846afcdf2e7a618d8b5" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 43283 } }
I20260808 04:06:55.625703 23393 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ce7014684154846afcdf2e7a618d8b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:06:55.625885 23393 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ce7014684154846afcdf2e7a618d8b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:06:55.626086 23393 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ce7014684154846afcdf2e7a618d8b5 [term 0 FOLLOWER]: Advancing to term 1
Aug 08 04:06:55 dist-test-slave-1l3l krb5kdc[22611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:06:55.628679 23393 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ce7014684154846afcdf2e7a618d8b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ce7014684154846afcdf2e7a618d8b5" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 43283 } }
I20260808 04:06:55.629297 23393 leader_election.cc:302] T 00000000000000000000000000000000 P 4ce7014684154846afcdf2e7a618d8b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ce7014684154846afcdf2e7a618d8b5; no voters:
I20260808 04:06:55.630237 23393 leader_election.cc:288] T 00000000000000000000000000000000 P 4ce7014684154846afcdf2e7a618d8b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:06:55.630434 23400 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ce7014684154846afcdf2e7a618d8b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:06:55.633041 23400 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ce7014684154846afcdf2e7a618d8b5 [term 1 LEADER]: Becoming Leader. State: Replica: 4ce7014684154846afcdf2e7a618d8b5, State: Running, Role: LEADER
I20260808 04:06:55.633651 23400 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ce7014684154846afcdf2e7a618d8b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ce7014684154846afcdf2e7a618d8b5" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 43283 } }
I20260808 04:06:55.638234 23397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:06:55.597538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56004 (local address 127.5.83.62:43283)
0808 04:06:55.612289 (+ 14751us) server_negotiation.cc:206] Beginning negotiation
0808 04:06:55.612317 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0808 04:06:55.612380 (+ 63us) server_negotiation.cc:407] Connection header received
0808 04:06:55.614876 (+ 2496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:06:55.614906 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:06:55.615437 (+ 531us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:06:55.615896 (+ 459us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:06:55.617545 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:06:55.619695 (+ 2150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:06:55.621592 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:06:55.622720 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:06:55.622917 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:06:55.627908 (+ 4991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:06:55.627920 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:06:55.627938 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:06:55.627982 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:06:55.632053 (+ 4071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:06:55.632975 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:06:55.632984 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:06:55.632994 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:06:55.633235 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:06:55.633739 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:06:55.633745 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:06:55.633749 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:06:55.634514 (+ 765us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:06:55.634825 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0808 04:06:55.636385 (+ 1560us) server_negotiation.cc:299] Negotiation successful
0808 04:06:55.636826 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14448,"thread_start_us":14938,"threads_started":1}
I20260808 04:06:55.645443 23403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ce7014684154846afcdf2e7a618d8b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ce7014684154846afcdf2e7a618d8b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ce7014684154846afcdf2e7a618d8b5" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 43283 } } }
I20260808 04:06:55.645869 23393 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ce7014684154846afcdf2e7a618d8b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:06:55.645962 23403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ce7014684154846afcdf2e7a618d8b5 [sys.catalog]: This master's current role is: LEADER
I20260808 04:06:55.647137 23401 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ce7014684154846afcdf2e7a618d8b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ce7014684154846afcdf2e7a618d8b5. Latest consensus state: current_term: 1 leader_uuid: "4ce7014684154846afcdf2e7a618d8b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ce7014684154846afcdf2e7a618d8b5" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 43283 } } }
I20260808 04:06:55.647967 23401 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ce7014684154846afcdf2e7a618d8b5 [sys.catalog]: This master's current role is: LEADER
I20260808 04:06:55.650795 23411 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:06:55.657032 23411 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:06:55.666539 23411 catalog_manager.cc:1347] Generated new cluster ID: aa333ec87beb46ae9d119d331859414b
I20260808 04:06:55.666651 23411 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:06:55.671239 23393 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260808 04:06:55.713296 23411 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:06:55.715576 23411 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:06:55.723685 23411 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ce7014684154846afcdf2e7a618d8b5: Generated new TSK 0
I20260808 04:06:55.724757 23411 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.83.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.1.keytab.
Entry for principal kudu/127.5.83.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.1.keytab.
I20260808 04:06:56.411242 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.83.20:59983/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.1:0
--local_ip_for_outbound_sockets=127.5.83.1
--webserver_interface=127.5.83.1
--webserver_port=0
--tserver_master_addrs=127.5.83.62:43283
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.1.keytab
--principal=kudu/127.5.83.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:35565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 04:06:56 dist-test-slave-1l3l krb5kdc[22611](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786162016, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260808 04:06:56.789382 23461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:06:56.789688 23461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:06:56.789830 23461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:06:56.797111 23461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:06:56.797217 23461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:06:56.797376 23461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.1
I20260808 04:06:56.808372 23461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:35565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.5.83.20:59983/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.1.keytab
--principal=kudu/127.5.83.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.83.1
--webserver_port=0
--tserver_master_addrs=127.5.83.62:43283
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.1
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:06:56.809612 23461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:06:56.811184 23461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:06:56.827083 23478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:06:56.831264 23477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:06:56.834342 23484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:06:56.835227 23461 server_base.cc:1034] running on GCE node
I20260808 04:06:56.835887 23461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:06:56.838953 23461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:06:56.840214 23461 hybrid_clock.cc:648] HybridClock initialized: now 1786162016840151 us; error 49 us; skew 500 ppm
I20260808 04:06:56.840538 23461 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:06:56 dist-test-slave-1l3l krb5kdc[22611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162016, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:06:56.843818 23461 init.cc:377] Logged in from keytab as kudu/127.5.83.1@KRBTEST.COM (short username kudu)
I20260808 04:06:56.845520 23461 webserver.cc:460] Webserver started at http://127.5.83.1:38053/ using document root <none> and password file <none>
I20260808 04:06:56.846280 23461 fs_manager.cc:359] Metadata directory not provided
I20260808 04:06:56.846402 23461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:06:56.846701 23461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 04:06:56 dist-test-slave-1l3l krb5kdc[22611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162016, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.1@KRBTEST.COM for HTTP/127.5.83.20@KRBTEST.COM
I20260808 04:06:56.867688 23461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-0/data/instance:
uuid: "e743b97eff7f41cabd30b29116a1be2c"
format_stamp: "Formatted at 2026-08-08 04:06:56 on dist-test-slave-1l3l"
server_key: "84ec83f66300efa909671dacd265a330"
server_key_iv: "b3f0c4d42f7088bdb9f9165aeb5164c4"
server_key_version: "kuduclusterkey@0"
I20260808 04:06:56.868886 23461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-0/wal/instance:
uuid: "e743b97eff7f41cabd30b29116a1be2c"
format_stamp: "Formatted at 2026-08-08 04:06:56 on dist-test-slave-1l3l"
server_key: "84ec83f66300efa909671dacd265a330"
server_key_iv: "b3f0c4d42f7088bdb9f9165aeb5164c4"
server_key_version: "kuduclusterkey@0"
I20260808 04:06:56.877496 23461 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.000s
I20260808 04:06:56.928900 23492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:06:56.931078 23461 fs_manager.cc:725] Time spent opening block manager: real 0.029s user 0.012s sys 0.016s
I20260808 04:06:56.931354 23461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-0/wal
uuid: "e743b97eff7f41cabd30b29116a1be2c"
format_stamp: "Formatted at 2026-08-08 04:06:56 on dist-test-slave-1l3l"
server_key: "84ec83f66300efa909671dacd265a330"
server_key_iv: "b3f0c4d42f7088bdb9f9165aeb5164c4"
server_key_version: "kuduclusterkey@0"
I20260808 04:06:56.931615 23461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:06:56.984704 23461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:06:56.989799 23461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:06:56.990192 23461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:06:56.991492 23461 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:06:56.993439 23461 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:06:56.993578 23461 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260808 04:06:56.993767 23461 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:06:56.993884 23461 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:06:57.076951 23666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.1:45467 every 8 connection(s)
I20260808 04:06:57.076967 23461 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.1:45467
I20260808 04:06:57.080091 23461 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
I20260808 04:06:57.081007 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 23461
I20260808 04:06:57.081230 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-0/wal/instance
I20260808 04:06:57.082132 5452 external_mini_cluster.cc:1428] Setting key aec6a9dc492ac583234d3786f84f891a
Aug 08 04:06:57 dist-test-slave-1l3l krb5kdc[22611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162016, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.1@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:06:57.176157 23671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:06:57.086350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.1:54979 (local address 127.5.83.62:43283)
0808 04:06:57.129829 (+ 43479us) server_negotiation.cc:206] Beginning negotiation
0808 04:06:57.129837 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:06:57.130993 (+ 1156us) server_negotiation.cc:407] Connection header received
0808 04:06:57.141067 (+ 10074us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:06:57.141074 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:06:57.141189 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:06:57.141282 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:06:57.148109 (+ 6827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:06:57.149285 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:06:57.158621 (+ 9336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:06:57.159280 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:06:57.159416 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:06:57.169286 (+ 9870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:06:57.169295 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:06:57.169299 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:06:57.169358 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:06:57.172568 (+ 3210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:06:57.173467 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:06:57.173473 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:06:57.173477 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:06:57.173639 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:06:57.174132 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:06:57.174137 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:06:57.174141 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:06:57.174495 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:06:57.174639 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0808 04:06:57.175503 (+ 864us) server_negotiation.cc:299] Negotiation successful
0808 04:06:57.175758 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43310,"thread_start_us":43489,"threads_started":1}
I20260808 04:06:57.177381 23670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:06:57.084598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:43283 (local address 127.5.83.1:54979)
0808 04:06:57.130736 (+ 46138us) negotiation.cc:107] Waiting for socket to connect
0808 04:06:57.130810 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0808 04:06:57.134787 (+ 3977us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:06:57.141922 (+ 7135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:06:57.141935 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:06:57.142649 (+ 714us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:06:57.147882 (+ 5233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:06:57.147911 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:06:57.156244 (+ 8333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:06:57.156251 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:06:57.158431 (+ 2180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:06:57.158462 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:06:57.164820 (+ 6358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:06:57.164829 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:06:57.165148 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:06:57.166087 (+ 939us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:06:57.166121 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:06:57.168997 (+ 2876us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:06:57.172701 (+ 3704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:06:57.172711 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:06:57.172729 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:06:57.173306 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:06:57.173750 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:06:57.173756 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:06:57.173760 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:06:57.174015 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:06:57.174630 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:06:57.174639 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:06:57.175047 (+ 408us) client_negotiation.cc:769] Sending connection context
0808 04:06:57.175388 (+ 341us) client_negotiation.cc:240] Negotiation successful
0808 04:06:57.175959 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45735,"thread_start_us":52335,"threads_started":1}
WARNING: no policy specified for kudu/127.5.83.2@KRBTEST.COM; defaulting to no policy
I20260808 04:06:57.179431 23667 heartbeater.cc:347] Connected to a master server at 127.5.83.62:43283
I20260808 04:06:57.179817 23667 heartbeater.cc:464] Registering TS with master...
I20260808 04:06:57.180780 23667 heartbeater.cc:511] Master 127.5.83.62:43283 requested a full tablet report, sending...
I20260808 04:06:57.183260 23336 ts_manager.cc:194] Registered new tserver with Master: e743b97eff7f41cabd30b29116a1be2c (127.5.83.1:45467)
I20260808 04:06:57.185266 23336 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.1@KRBTEST.COM'} at 127.5.83.1:54979
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.2@KRBTEST.COM" created.
Aug 08 04:06:57 dist-test-slave-1l3l krb5kdc[22611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162016, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.62@KRBTEST.COM for HTTP/127.5.83.20@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.2.keytab.
Entry for principal kudu/127.5.83.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.2.keytab.
W20260808 04:06:57.227844 5452 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.5.83.2: real 0.140s user 0.002s sys 0.019s
I20260808 04:06:57.228660 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.83.20:59983/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.2:0
--local_ip_for_outbound_sockets=127.5.83.2
--webserver_interface=127.5.83.2
--webserver_port=0
--tserver_master_addrs=127.5.83.62:43283
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.2.keytab
--principal=kudu/127.5.83.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:35565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 04:06:57 dist-test-slave-1l3l krb5kdc[22611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162016, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.62@KRBTEST.COM for HTTP/127.5.83.20@KRBTEST.COM
W20260808 04:06:57.638903 23674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:06:57.639330 23674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:06:57.639604 23674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:06:57.650758 23674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:06:57.650903 23674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:06:57.651084 23674 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.2
I20260808 04:06:57.666363 23674 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:35565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.5.83.20:59983/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.2.keytab
--principal=kudu/127.5.83.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.83.2
--webserver_port=0
--tserver_master_addrs=127.5.83.62:43283
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.2
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:06:57.668143 23674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:06:57.669883 23674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:06:57.694473 23687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:06:57.694453 23685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:06:57.703902 23689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:06:57.708599 23674 server_base.cc:1034] running on GCE node
I20260808 04:06:57.709237 23674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:06:57.713050 23674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:06:57.714395 23674 hybrid_clock.cc:648] HybridClock initialized: now 1786162017714230 us; error 45 us; skew 500 ppm
I20260808 04:06:57.714856 23674 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:06:57 dist-test-slave-1l3l krb5kdc[22611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162017, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:06:57.718705 23674 init.cc:377] Logged in from keytab as kudu/127.5.83.2@KRBTEST.COM (short username kudu)
I20260808 04:06:57.727159 23674 webserver.cc:460] Webserver started at http://127.5.83.2:46679/ using document root <none> and password file <none>
I20260808 04:06:57.728224 23674 fs_manager.cc:359] Metadata directory not provided
I20260808 04:06:57.728394 23674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:06:57.728780 23674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 04:06:57 dist-test-slave-1l3l krb5kdc[22611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162017, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.2@KRBTEST.COM for HTTP/127.5.83.20@KRBTEST.COM
I20260808 04:06:57.746750 23674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-1/data/instance:
uuid: "64fedc092b874361b91f71701d089bbd"
format_stamp: "Formatted at 2026-08-08 04:06:57 on dist-test-slave-1l3l"
server_key: "1eb1fdabdd3cb84d6feee2e4f44d7c4e"
server_key_iv: "2fda99a8202226bda1717cb7a4dda41c"
server_key_version: "kuduclusterkey@0"
I20260808 04:06:57.747839 23674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-1/wal/instance:
uuid: "64fedc092b874361b91f71701d089bbd"
format_stamp: "Formatted at 2026-08-08 04:06:57 on dist-test-slave-1l3l"
server_key: "1eb1fdabdd3cb84d6feee2e4f44d7c4e"
server_key_iv: "2fda99a8202226bda1717cb7a4dda41c"
server_key_version: "kuduclusterkey@0"
I20260808 04:06:57.768335 23674 fs_manager.cc:691] Time spent creating directory manager: real 0.020s user 0.010s sys 0.009s
I20260808 04:06:57.800244 23699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:06:57.807327 23674 fs_manager.cc:725] Time spent opening block manager: real 0.025s user 0.007s sys 0.009s
I20260808 04:06:57.807565 23674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-1/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-1/wal
uuid: "64fedc092b874361b91f71701d089bbd"
format_stamp: "Formatted at 2026-08-08 04:06:57 on dist-test-slave-1l3l"
server_key: "1eb1fdabdd3cb84d6feee2e4f44d7c4e"
server_key_iv: "2fda99a8202226bda1717cb7a4dda41c"
server_key_version: "kuduclusterkey@0"
I20260808 04:06:57.807904 23674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:06:57.859025 23674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:06:57.862982 23674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:06:57.863274 23674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:06:57.864318 23674 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:06:57.865710 23674 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:06:57.865789 23674 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:06:57.865914 23674 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:06:57.865972 23674 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:06:57.904402 23674 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.2:46461
I20260808 04:06:57.904413 23879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.2:46461 every 8 connection(s)
I20260808 04:06:57.905666 23674 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
I20260808 04:06:57.910482 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 23674
I20260808 04:06:57.910635 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-1/wal/instance
I20260808 04:06:57.911055 5452 external_mini_cluster.cc:1428] Setting key 349bd781f716926745c4c8cede675664
Aug 08 04:06:57 dist-test-slave-1l3l krb5kdc[22611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162017, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.2@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:06:57.928432 23884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:06:57.908325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.2:35899 (local address 127.5.83.62:43283)
0808 04:06:57.908838 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0808 04:06:57.908844 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:06:57.909750 (+ 906us) server_negotiation.cc:407] Connection header received
0808 04:06:57.912281 (+ 2531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:06:57.912287 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:06:57.912381 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:06:57.912505 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:06:57.914341 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:06:57.915534 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:06:57.917881 (+ 2347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:06:57.918444 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:06:57.918562 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:06:57.922601 (+ 4039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:06:57.922609 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:06:57.922613 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:06:57.922654 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:06:57.925541 (+ 2887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:06:57.926210 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:06:57.926214 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:06:57.926216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:06:57.926344 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:06:57.926751 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:06:57.926754 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:06:57.926756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:06:57.927024 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:06:57.927156 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0808 04:06:57.927838 (+ 682us) server_negotiation.cc:299] Negotiation successful
0808 04:06:57.928063 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":260,"threads_started":1}
I20260808 04:06:57.929304 23885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:06:57.908743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:43283 (local address 127.5.83.2:35899)
0808 04:06:57.909570 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0808 04:06:57.909620 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0808 04:06:57.911972 (+ 2352us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:06:57.912744 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:06:57.912755 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:06:57.913358 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:06:57.914124 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:06:57.914151 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:06:57.915734 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:06:57.915739 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:06:57.917720 (+ 1981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:06:57.917735 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:06:57.918674 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:06:57.918681 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:06:57.918953 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:06:57.919730 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:06:57.919761 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:06:57.922354 (+ 2593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:06:57.925675 (+ 3321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:06:57.925684 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:06:57.925698 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:06:57.926085 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:06:57.926448 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:06:57.926453 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:06:57.926456 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:06:57.926637 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:06:57.927145 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:06:57.927152 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:06:57.927459 (+ 307us) client_negotiation.cc:769] Sending connection context
0808 04:06:57.927689 (+ 230us) client_negotiation.cc:240] Negotiation successful
0808 04:06:57.928194 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":295,"threads_started":1}
I20260808 04:06:57.930687 23881 heartbeater.cc:347] Connected to a master server at 127.5.83.62:43283
I20260808 04:06:57.930998 23881 heartbeater.cc:464] Registering TS with master...
I20260808 04:06:57.931756 23881 heartbeater.cc:511] Master 127.5.83.62:43283 requested a full tablet report, sending...
I20260808 04:06:57.933194 23336 ts_manager.cc:194] Registered new tserver with Master: 64fedc092b874361b91f71701d089bbd (127.5.83.2:46461)
I20260808 04:06:57.933970 23336 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.2@KRBTEST.COM'} at 127.5.83.2:35899
WARNING: no policy specified for kudu/127.5.83.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.3.keytab.
Entry for principal kudu/127.5.83.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.3.keytab.
I20260808 04:06:57.976917 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.83.20:59983/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.3:0
--local_ip_for_outbound_sockets=127.5.83.3
--webserver_interface=127.5.83.3
--webserver_port=0
--tserver_master_addrs=127.5.83.62:43283
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.3.keytab
--principal=kudu/127.5.83.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:35565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:06:58.188902 23667 heartbeater.cc:503] Master 127.5.83.62:43283 was elected leader, sending a full tablet report...
W20260808 04:06:58.315337 23888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:06:58.315644 23888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:06:58.315763 23888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:06:58.322597 23888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:06:58.322695 23888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:06:58.322805 23888 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.3
I20260808 04:06:58.332238 23888 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:35565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.5.83.20:59983/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.3.keytab
--principal=kudu/127.5.83.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.83.3
--webserver_port=0
--tserver_master_addrs=127.5.83.62:43283
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.3
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:06:58.333451 23888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:06:58.334700 23888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:06:58.347011 23899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:06:58.347251 23897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:06:58.347030 23902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:06:58.348469 23888 server_base.cc:1034] running on GCE node
I20260808 04:06:58.349005 23888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:06:58.349862 23888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:06:58.351054 23888 hybrid_clock.cc:648] HybridClock initialized: now 1786162018351004 us; error 43 us; skew 500 ppm
I20260808 04:06:58.351338 23888 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:06:58 dist-test-slave-1l3l krb5kdc[22611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162018, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:06:58.354385 23888 init.cc:377] Logged in from keytab as kudu/127.5.83.3@KRBTEST.COM (short username kudu)
I20260808 04:06:58.355690 23888 webserver.cc:460] Webserver started at http://127.5.83.3:44583/ using document root <none> and password file <none>
I20260808 04:06:58.356338 23888 fs_manager.cc:359] Metadata directory not provided
I20260808 04:06:58.356449 23888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:06:58.356736 23888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 04:06:58 dist-test-slave-1l3l krb5kdc[22611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162018, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.3@KRBTEST.COM for HTTP/127.5.83.20@KRBTEST.COM
I20260808 04:06:58.372453 23888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-2/data/instance:
uuid: "f64e5a37850f468b94f506a69e9baa00"
format_stamp: "Formatted at 2026-08-08 04:06:58 on dist-test-slave-1l3l"
server_key: "17c4b602b1316b8410f8b9216b5ddd6b"
server_key_iv: "d056a41fef3096fd4fb6f7c6c957875f"
server_key_version: "kuduclusterkey@0"
I20260808 04:06:58.373281 23888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-2/wal/instance:
uuid: "f64e5a37850f468b94f506a69e9baa00"
format_stamp: "Formatted at 2026-08-08 04:06:58 on dist-test-slave-1l3l"
server_key: "17c4b602b1316b8410f8b9216b5ddd6b"
server_key_iv: "d056a41fef3096fd4fb6f7c6c957875f"
server_key_version: "kuduclusterkey@0"
I20260808 04:06:58.378476 23888 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 04:06:58.394235 23916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:06:58.395679 23888 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260808 04:06:58.395861 23888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-2/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-2/wal
uuid: "f64e5a37850f468b94f506a69e9baa00"
format_stamp: "Formatted at 2026-08-08 04:06:58 on dist-test-slave-1l3l"
server_key: "17c4b602b1316b8410f8b9216b5ddd6b"
server_key_iv: "d056a41fef3096fd4fb6f7c6c957875f"
server_key_version: "kuduclusterkey@0"
I20260808 04:06:58.396065 23888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:06:58.424451 23888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:06:58.428267 23888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:06:58.428544 23888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:06:58.429522 23888 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:06:58.430868 23888 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:06:58.430943 23888 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:06:58.431079 23888 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:06:58.431151 23888 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:06:58.461392 23888 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.3:33939
I20260808 04:06:58.461431 24108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.3:33939 every 8 connection(s)
I20260808 04:06:58.462612 23888 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
I20260808 04:06:58.464541 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 23888
I20260808 04:06:58.464708 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-2/wal/instance
I20260808 04:06:58.465211 5452 external_mini_cluster.cc:1428] Setting key 3dee9c289b1b41ae3ad2930b4177f741
Aug 08 04:06:58 dist-test-slave-1l3l krb5kdc[22611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162018, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.3@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:06:58.482044 24112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:06:58.465169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.3:47345 (local address 127.5.83.62:43283)
0808 04:06:58.468107 (+ 2938us) server_negotiation.cc:206] Beginning negotiation
0808 04:06:58.468112 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:06:58.468129 (+ 17us) server_negotiation.cc:407] Connection header received
0808 04:06:58.468491 (+ 362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:06:58.468497 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:06:58.468578 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:06:58.468672 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:06:58.470720 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:06:58.471510 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:06:58.472989 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:06:58.473459 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:06:58.473559 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:06:58.476639 (+ 3080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:06:58.476644 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:06:58.476647 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:06:58.476678 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:06:58.479083 (+ 2405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:06:58.479692 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:06:58.479696 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:06:58.479698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:06:58.479845 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:06:58.480213 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:06:58.480216 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:06:58.480218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:06:58.480509 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:06:58.480637 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0808 04:06:58.481460 (+ 823us) server_negotiation.cc:299] Negotiation successful
0808 04:06:58.481662 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2822,"thread_start_us":2674,"threads_started":1}
I20260808 04:06:58.482755 24114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:06:58.465745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:43283 (local address 127.5.83.3:47345)
0808 04:06:58.466531 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0808 04:06:58.466575 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 04:06:58.468200 (+ 1625us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:06:58.469257 (+ 1057us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:06:58.469265 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:06:58.469818 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:06:58.470563 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:06:58.470581 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:06:58.471629 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:06:58.471633 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:06:58.472849 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:06:58.472861 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:06:58.473634 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:06:58.473639 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:06:58.473813 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:06:58.474413 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:06:58.474434 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:06:58.476429 (+ 1995us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:06:58.479194 (+ 2765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:06:58.479200 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:06:58.479210 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:06:58.479602 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:06:58.479944 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:06:58.479947 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:06:58.479949 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:06:58.480130 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:06:58.480645 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:06:58.480652 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:06:58.480965 (+ 313us) client_negotiation.cc:769] Sending connection context
0808 04:06:58.481180 (+ 215us) client_negotiation.cc:240] Negotiation successful
0808 04:06:58.481662 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":339,"threads_started":1}
I20260808 04:06:58.483961 24110 heartbeater.cc:347] Connected to a master server at 127.5.83.62:43283
I20260808 04:06:58.484232 24110 heartbeater.cc:464] Registering TS with master...
I20260808 04:06:58.484866 24110 heartbeater.cc:511] Master 127.5.83.62:43283 requested a full tablet report, sending...
I20260808 04:06:58.486148 23336 ts_manager.cc:194] Registered new tserver with Master: f64e5a37850f468b94f506a69e9baa00 (127.5.83.3:33939)
I20260808 04:06:58.486920 23336 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.3@KRBTEST.COM'} at 127.5.83.3:47345
I20260808 04:06:58.494715 5452 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:06:58.496522 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 23461
Aug 08 04:06:58 dist-test-slave-1l3l krb5kdc[22611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.1@KRBTEST.COM
I20260808 04:06:58.511845 24116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:06:58.497403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53298 (local address 127.5.83.1:45467)
0808 04:06:58.497993 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0808 04:06:58.497999 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:06:58.498021 (+ 22us) server_negotiation.cc:407] Connection header received
0808 04:06:58.498080 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:06:58.498085 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:06:58.498235 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:06:58.498351 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:06:58.499749 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:06:58.500840 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:06:58.502361 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:06:58.502857 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:06:58.502993 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:06:58.506944 (+ 3951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:06:58.506954 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:06:58.506965 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:06:58.506996 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:06:58.509579 (+ 2583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:06:58.510185 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:06:58.510191 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:06:58.510193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:06:58.510329 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:06:58.510580 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:06:58.510583 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:06:58.510585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:06:58.510874 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:06:58.511088 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0808 04:06:58.511264 (+ 176us) server_negotiation.cc:299] Negotiation successful
0808 04:06:58.511478 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":304,"threads_started":1}
I20260808 04:06:58.513001 23659 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:06:58.597023 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 23461
I20260808 04:06:58.615409 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 23674
Aug 08 04:06:58 dist-test-slave-1l3l krb5kdc[22611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.2@KRBTEST.COM
I20260808 04:06:58.630820 24120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:06:58.616315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43912 (local address 127.5.83.2:46461)
0808 04:06:58.616896 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0808 04:06:58.616921 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0808 04:06:58.616948 (+ 27us) server_negotiation.cc:407] Connection header received
0808 04:06:58.617007 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:06:58.617013 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:06:58.617223 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:06:58.617345 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:06:58.618495 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:06:58.619475 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:06:58.620896 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:06:58.621430 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:06:58.621523 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:06:58.625524 (+ 4001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:06:58.625536 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:06:58.625550 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:06:58.625591 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:06:58.628361 (+ 2770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:06:58.629081 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:06:58.629087 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:06:58.629090 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:06:58.629225 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:06:58.629517 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:06:58.629520 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:06:58.629522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:06:58.629846 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:06:58.630072 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0808 04:06:58.630256 (+ 184us) server_negotiation.cc:299] Negotiation successful
0808 04:06:58.630448 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":315,"threads_started":1}
I20260808 04:06:58.632112 23873 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:06:58.716488 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 23674
I20260808 04:06:58.734670 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 23888
Aug 08 04:06:58 dist-test-slave-1l3l krb5kdc[22611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.3@KRBTEST.COM
I20260808 04:06:58.750837 24122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:06:58.735653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51068 (local address 127.5.83.3:33939)
0808 04:06:58.736338 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0808 04:06:58.736345 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:06:58.736374 (+ 29us) server_negotiation.cc:407] Connection header received
0808 04:06:58.736456 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:06:58.736463 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:06:58.736666 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:06:58.736810 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:06:58.738201 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:06:58.739200 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:06:58.740650 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:06:58.741188 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:06:58.741303 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:06:58.745396 (+ 4093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:06:58.745428 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:06:58.745441 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:06:58.745479 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:06:58.748183 (+ 2704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:06:58.748826 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:06:58.748832 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:06:58.748834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:06:58.748964 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:06:58.749306 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:06:58.749309 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:06:58.749311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:06:58.749673 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:06:58.749982 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0808 04:06:58.750256 (+ 274us) server_negotiation.cc:299] Negotiation successful
0808 04:06:58.750521 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":341,"threads_started":1}
I20260808 04:06:58.752066 24102 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:06:58.832502 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 23888
I20260808 04:06:58.849929 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 23287
I20260808 04:06:58.851182 23387 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:06:58.956192 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 23287
I20260808 04:06:58.981436 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:43283
--webserver_interface=127.5.83.62
--webserver_port=46183
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab
--principal=kudu/127.5.83.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:35565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.62:43283
--ranger_config_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.83.20:59983/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.5.83.20:59983/kms with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:06:59.300369 24127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:06:59.300630 24127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:06:59.300724 24127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:06:59.307601 24127 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:06:59.307694 24127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:06:59.307777 24127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:06:59.307848 24127 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:06:59.307910 24127 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:06:59.317621 24127 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:35565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.5.83.20:59983/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.5.83.62:43283
--ranger_config_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab
--principal=kudu/127.5.83.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:43283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.62
--webserver_port=46183
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:06:59.318815 24127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:06:59.320097 24127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:06:59.331045 24135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:06:59.331099 24143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:06:59.331103 24134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:06:59.332078 24127 server_base.cc:1034] running on GCE node
I20260808 04:06:59.332762 24127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:06:59.333992 24127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:06:59.335188 24127 hybrid_clock.cc:648] HybridClock initialized: now 1786162019335140 us; error 39 us; skew 500 ppm
I20260808 04:06:59.335520 24127 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:06:59 dist-test-slave-1l3l krb5kdc[22611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162019, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:06:59.338470 24127 init.cc:377] Logged in from keytab as kudu/127.5.83.62@KRBTEST.COM (short username kudu)
I20260808 04:06:59.339857 24127 webserver.cc:460] Webserver started at http://127.5.83.62:46183/ using document root <none> and password file <none>
I20260808 04:06:59.340441 24127 fs_manager.cc:359] Metadata directory not provided
I20260808 04:06:59.340569 24127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 08 04:06:59 dist-test-slave-1l3l krb5kdc[22611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162019, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.62@KRBTEST.COM for HTTP/127.5.83.20@KRBTEST.COM
I20260808 04:06:59.382170 24127 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 04:06:59.385933 24161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:06:59.387490 24127 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 04:06:59.387689 24127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/master-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/master-0/wal
uuid: "4ce7014684154846afcdf2e7a618d8b5"
format_stamp: "Formatted at 2026-08-08 04:06:55 on dist-test-slave-1l3l"
server_key: "4131bc8af29cb25a8d38e2d5fc20cb69"
server_key_iv: "268bb724d13f17d31028f72cf1f2eee9"
server_key_version: "kuduclusterkey@0"
I20260808 04:06:59.389042 24127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:06:59.417357 24127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:06:59.421459 24127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:06:59.421697 24127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:06:59.438495 24127 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.62:43283
I20260808 04:06:59.438521 24242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.62:43283 every 8 connection(s)
I20260808 04:06:59.439952 24127 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
I20260808 04:06:59.444305 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 24127
I20260808 04:06:59.445114 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.83.20:59983/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.1:45467
--local_ip_for_outbound_sockets=127.5.83.1
--tserver_master_addrs=127.5.83.62:43283
--webserver_port=38053
--webserver_interface=127.5.83.1
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.1.keytab
--principal=kudu/127.5.83.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:35565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.5.83.20:59983/kms with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:06:59.446509 24244 sys_catalog.cc:263] Verifying existing consensus state
I20260808 04:06:59.450170 24244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ce7014684154846afcdf2e7a618d8b5: Bootstrap starting.
I20260808 04:06:59.468487 24244 log.cc:824] T 00000000000000000000000000000000 P 4ce7014684154846afcdf2e7a618d8b5: Log is configured to *not* fsync() on all Append() calls
I20260808 04:06:59.476927 24244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ce7014684154846afcdf2e7a618d8b5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 04:06:59.477398 24244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ce7014684154846afcdf2e7a618d8b5: Bootstrap complete.
I20260808 04:06:59.485700 24244 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ce7014684154846afcdf2e7a618d8b5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ce7014684154846afcdf2e7a618d8b5" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 43283 } }
I20260808 04:06:59.486503 24244 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ce7014684154846afcdf2e7a618d8b5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ce7014684154846afcdf2e7a618d8b5, State: Initialized, Role: FOLLOWER
I20260808 04:06:59.487062 24244 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ce7014684154846afcdf2e7a618d8b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ce7014684154846afcdf2e7a618d8b5" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 43283 } }
I20260808 04:06:59.487267 24244 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ce7014684154846afcdf2e7a618d8b5 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:06:59.487370 24244 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ce7014684154846afcdf2e7a618d8b5 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:06:59.487540 24244 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ce7014684154846afcdf2e7a618d8b5 [term 1 FOLLOWER]: Advancing to term 2
I20260808 04:06:59.491071 24244 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ce7014684154846afcdf2e7a618d8b5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ce7014684154846afcdf2e7a618d8b5" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 43283 } }
I20260808 04:06:59.491520 24244 leader_election.cc:302] T 00000000000000000000000000000000 P 4ce7014684154846afcdf2e7a618d8b5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ce7014684154846afcdf2e7a618d8b5; no voters:
I20260808 04:06:59.492112 24244 leader_election.cc:288] T 00000000000000000000000000000000 P 4ce7014684154846afcdf2e7a618d8b5 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 04:06:59.492242 24253 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ce7014684154846afcdf2e7a618d8b5 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 04:06:59.493314 24253 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ce7014684154846afcdf2e7a618d8b5 [term 2 LEADER]: Becoming Leader. State: Replica: 4ce7014684154846afcdf2e7a618d8b5, State: Running, Role: LEADER
I20260808 04:06:59.493887 24253 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ce7014684154846afcdf2e7a618d8b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ce7014684154846afcdf2e7a618d8b5" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 43283 } }
I20260808 04:06:59.494159 24244 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ce7014684154846afcdf2e7a618d8b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:06:59.496338 24256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ce7014684154846afcdf2e7a618d8b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4ce7014684154846afcdf2e7a618d8b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ce7014684154846afcdf2e7a618d8b5" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 43283 } } }
I20260808 04:06:59.496703 24256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ce7014684154846afcdf2e7a618d8b5 [sys.catalog]: This master's current role is: LEADER
I20260808 04:06:59.501685 24258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ce7014684154846afcdf2e7a618d8b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ce7014684154846afcdf2e7a618d8b5. Latest consensus state: current_term: 2 leader_uuid: "4ce7014684154846afcdf2e7a618d8b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ce7014684154846afcdf2e7a618d8b5" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 43283 } } }
I20260808 04:06:59.502529 24258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ce7014684154846afcdf2e7a618d8b5 [sys.catalog]: This master's current role is: LEADER
I20260808 04:06:59.504487 24263 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:06:59.510247 24263 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:06:59.511590 24263 catalog_manager.cc:1259] Loaded cluster ID: aa333ec87beb46ae9d119d331859414b
I20260808 04:06:59.511745 24263 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:06:59.515789 24263 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:06:59.517189 24263 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4ce7014684154846afcdf2e7a618d8b5: Loaded TSK: 0
I20260808 04:06:59.518747 24263 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:06:59.520256 24244 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260808 04:06:59.880779 24248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:06:59.881207 24248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:06:59.881415 24248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:06:59.891799 24248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:06:59.891978 24248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:06:59.892201 24248 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.1
I20260808 04:06:59.906759 24248 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:35565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.5.83.20:59983/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.1.keytab
--principal=kudu/127.5.83.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.1:45467
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.83.1
--webserver_port=38053
--tserver_master_addrs=127.5.83.62:43283
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.1
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:06:59.908090 24248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:06:59.909425 24248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:06:59.939128 24313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:06:59.939838 24312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:06:59.940014 24315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:06:59.941708 24248 server_base.cc:1034] running on GCE node
I20260808 04:06:59.942520 24248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:06:59.943786 24248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:06:59.945122 24248 hybrid_clock.cc:648] HybridClock initialized: now 1786162019945017 us; error 85 us; skew 500 ppm
I20260808 04:06:59.945611 24248 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:06:59 dist-test-slave-1l3l krb5kdc[22611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162019, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:06:59.951102 24248 init.cc:377] Logged in from keytab as kudu/127.5.83.1@KRBTEST.COM (short username kudu)
I20260808 04:06:59.953084 24248 webserver.cc:460] Webserver started at http://127.5.83.1:38053/ using document root <none> and password file <none>
I20260808 04:06:59.954083 24248 fs_manager.cc:359] Metadata directory not provided
I20260808 04:06:59.954313 24248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 08 04:06:59 dist-test-slave-1l3l krb5kdc[22611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162019, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.1@KRBTEST.COM for HTTP/127.5.83.20@KRBTEST.COM
I20260808 04:06:59.978859 24248 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 04:06:59.983911 24323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:06:59.985674 24248 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260808 04:06:59.985915 24248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-0/wal
uuid: "e743b97eff7f41cabd30b29116a1be2c"
format_stamp: "Formatted at 2026-08-08 04:06:56 on dist-test-slave-1l3l"
server_key: "84ec83f66300efa909671dacd265a330"
server_key_iv: "b3f0c4d42f7088bdb9f9165aeb5164c4"
server_key_version: "kuduclusterkey@0"
I20260808 04:06:59.987284 24248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:07:00.011853 24248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:07:00.017176 24248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:07:00.017462 24248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:07:00.018589 24248 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:07:00.020062 24248 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:07:00.020224 24248 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:07:00.020428 24248 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:07:00.020583 24248 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:07:00.060528 24248 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.1:45467
I20260808 04:07:00.060580 24487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.1:45467 every 8 connection(s)
I20260808 04:07:00.062040 24248 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
I20260808 04:07:00.063048 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 24248
I20260808 04:07:00.063901 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.83.20:59983/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.2:46461
--local_ip_for_outbound_sockets=127.5.83.2
--tserver_master_addrs=127.5.83.62:43283
--webserver_port=46679
--webserver_interface=127.5.83.2
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.2.keytab
--principal=kudu/127.5.83.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:35565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.5.83.20:59983/kms with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 04:07:00 dist-test-slave-1l3l krb5kdc[22611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162019, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.1@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:07:00.116418 24495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:00.068243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:43283 (local address 127.5.83.1:57987)
0808 04:07:00.076003 (+ 7760us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:00.076062 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0808 04:07:00.079899 (+ 3837us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:07:00.089171 (+ 9272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:07:00.089185 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:07:00.090221 (+ 1036us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:07:00.091071 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:07:00.091103 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:07:00.097320 (+ 6217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:07:00.097336 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:07:00.099374 (+ 2038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:07:00.099402 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:07:00.100872 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:07:00.100880 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:07:00.101169 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:00.103332 (+ 2163us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:07:00.103366 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:07:00.106947 (+ 3581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:07:00.111396 (+ 4449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:07:00.111405 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:07:00.111420 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:07:00.112073 (+ 653us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:07:00.112601 (+ 528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:07:00.112606 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:07:00.112610 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:07:00.112849 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:07:00.113923 (+ 1074us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:07:00.113931 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:07:00.114314 (+ 383us) client_negotiation.cc:769] Sending connection context
0808 04:07:00.114583 (+ 269us) client_negotiation.cc:240] Negotiation successful
0808 04:07:00.115095 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":3350,"threads_started":1}
I20260808 04:07:00.117388 24496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:00.069428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.1:57987 (local address 127.5.83.62:43283)
0808 04:07:00.085470 (+ 16042us) server_negotiation.cc:206] Beginning negotiation
0808 04:07:00.085488 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0808 04:07:00.085537 (+ 49us) server_negotiation.cc:407] Connection header received
0808 04:07:00.085959 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:07:00.085979 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:07:00.088092 (+ 2113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:07:00.091396 (+ 3304us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:07:00.091933 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:00.097095 (+ 5162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:00.099560 (+ 2465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:00.100612 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:00.100851 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:00.107281 (+ 6430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:07:00.107292 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:07:00.107311 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:07:00.107373 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:07:00.111216 (+ 3843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:00.112210 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:00.112219 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:00.112227 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:00.112445 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:00.112995 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:00.113000 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:00.113003 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:00.113669 (+ 666us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:07:00.113839 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0808 04:07:00.115877 (+ 2038us) server_negotiation.cc:299] Negotiation successful
0808 04:07:00.116259 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15769,"thread_start_us":15573,"threads_started":1}
I20260808 04:07:00.120530 24490 heartbeater.cc:347] Connected to a master server at 127.5.83.62:43283
I20260808 04:07:00.120814 24490 heartbeater.cc:464] Registering TS with master...
I20260808 04:07:00.121502 24490 heartbeater.cc:511] Master 127.5.83.62:43283 requested a full tablet report, sending...
W20260808 04:07:00.124225 24490 heartbeater.cc:644] Failed to heartbeat to 127.5.83.62:43283 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260808 04:07:00.124473 24490 heartbeater.cc:464] Registering TS with master...
I20260808 04:07:00.124714 24490 heartbeater.cc:511] Master 127.5.83.62:43283 requested a full tablet report, sending...
I20260808 04:07:00.125720 24490 heartbeater.cc:464] Registering TS with master...
I20260808 04:07:00.125970 24490 heartbeater.cc:511] Master 127.5.83.62:43283 requested a full tablet report, sending...
W20260808 04:07:00.127034 24490 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 04:07:00.455531 24498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:07:00.456085 24498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:07:00.456710 24498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:07:00.464512 24498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:07:00.464605 24498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:07:00.464751 24498 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.2
I20260808 04:07:00.477460 24498 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:35565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.5.83.20:59983/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.2.keytab
--principal=kudu/127.5.83.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.2:46461
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.83.2
--webserver_port=46679
--tserver_master_addrs=127.5.83.62:43283
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.2
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:07:00.478677 24498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:07:00.480141 24498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:07:00.494782 24522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:07:00.494839 24523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:07:00.495963 24525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:07:00.496920 24498 server_base.cc:1034] running on GCE node
I20260808 04:07:00.497548 24498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:07:00.498651 24498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:07:00.499861 24498 hybrid_clock.cc:648] HybridClock initialized: now 1786162020499804 us; error 45 us; skew 500 ppm
I20260808 04:07:00.500154 24498 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:07:00 dist-test-slave-1l3l krb5kdc[22611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162020, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:07:00.510140 24498 init.cc:377] Logged in from keytab as kudu/127.5.83.2@KRBTEST.COM (short username kudu)
I20260808 04:07:00.511683 24498 webserver.cc:460] Webserver started at http://127.5.83.2:46679/ using document root <none> and password file <none>
I20260808 04:07:00.512406 24498 fs_manager.cc:359] Metadata directory not provided
I20260808 04:07:00.512542 24498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 08 04:07:00 dist-test-slave-1l3l krb5kdc[22611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162020, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.2@KRBTEST.COM for HTTP/127.5.83.20@KRBTEST.COM
I20260808 04:07:00.547392 24498 fs_manager.cc:709] Time spent opening directory manager: real 0.016s user 0.005s sys 0.009s
I20260808 04:07:00.558555 24540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:07:00.560134 24498 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.003s sys 0.008s
I20260808 04:07:00.560315 24498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-1/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-1/wal
uuid: "64fedc092b874361b91f71701d089bbd"
format_stamp: "Formatted at 2026-08-08 04:06:57 on dist-test-slave-1l3l"
server_key: "1eb1fdabdd3cb84d6feee2e4f44d7c4e"
server_key_iv: "2fda99a8202226bda1717cb7a4dda41c"
server_key_version: "kuduclusterkey@0"
I20260808 04:07:00.561324 24498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:07:00.610219 24498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:07:00.615635 24498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:07:00.616015 24498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:07:00.623435 24498 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:07:00.625427 24498 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:07:00.625555 24498 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:07:00.625669 24498 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:07:00.625746 24498 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:07:00.708066 24498 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.2:46461
I20260808 04:07:00.708539 24715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.2:46461 every 8 connection(s)
I20260808 04:07:00.709937 24498 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
I20260808 04:07:00.717005 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 24498
I20260808 04:07:00.717792 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.83.20:59983/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.3:33939
--local_ip_for_outbound_sockets=127.5.83.3
--tserver_master_addrs=127.5.83.62:43283
--webserver_port=44583
--webserver_interface=127.5.83.3
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.3.keytab
--principal=kudu/127.5.83.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:35565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.5.83.20:59983/kms with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 04:07:00 dist-test-slave-1l3l krb5kdc[22611](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786162020, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 08 04:07:00 dist-test-slave-1l3l krb5kdc[22611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162020, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.2@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:07:00.763206 24720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:00.715891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.2:48213 (local address 127.5.83.62:43283)
0808 04:07:00.724117 (+ 8226us) server_negotiation.cc:206] Beginning negotiation
0808 04:07:00.724125 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:07:00.727772 (+ 3647us) server_negotiation.cc:407] Connection header received
0808 04:07:00.729983 (+ 2211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:07:00.729989 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:07:00.730133 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:07:00.730213 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:07:00.734052 (+ 3839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:00.735280 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:00.737839 (+ 2559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:00.738514 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:00.738614 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:00.751252 (+ 12638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:07:00.751260 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:07:00.751265 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:07:00.751315 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:07:00.754893 (+ 3578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:00.760317 (+ 5424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:00.760324 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:00.760329 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:00.760506 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:00.761149 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:00.761155 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:00.761159 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:00.761527 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:07:00.761651 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0808 04:07:00.762609 (+ 958us) server_negotiation.cc:299] Negotiation successful
0808 04:07:00.762833 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8076,"thread_start_us":7940,"threads_started":1}
I20260808 04:07:00.765648 24724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:00.716491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:43283 (local address 127.5.83.2:48213)
0808 04:07:00.724774 (+ 8283us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:00.724843 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0808 04:07:00.727600 (+ 2757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:07:00.732220 (+ 4620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:07:00.732231 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:07:00.732991 (+ 760us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:07:00.733804 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:07:00.733830 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:07:00.735535 (+ 1705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:07:00.735540 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:07:00.737657 (+ 2117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:07:00.737672 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:07:00.738989 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:07:00.738996 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:07:00.739296 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:00.740085 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:07:00.740116 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:07:00.750914 (+ 10798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:07:00.759489 (+ 8575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:07:00.759500 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:07:00.759519 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:07:00.760169 (+ 650us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:07:00.760669 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:07:00.760674 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:07:00.760677 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:07:00.761019 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:07:00.761749 (+ 730us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:07:00.761757 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:07:00.762174 (+ 417us) client_negotiation.cc:769] Sending connection context
0808 04:07:00.763666 (+ 1492us) client_negotiation.cc:240] Negotiation successful
0808 04:07:00.764253 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7944,"thread_start_us":7699,"threads_started":1}
I20260808 04:07:00.769093 24716 heartbeater.cc:347] Connected to a master server at 127.5.83.62:43283
I20260808 04:07:00.769407 24716 heartbeater.cc:464] Registering TS with master...
I20260808 04:07:00.770238 24716 heartbeater.cc:511] Master 127.5.83.62:43283 requested a full tablet report, sending...
I20260808 04:07:00.772222 24187 ts_manager.cc:194] Registered new tserver with Master: 64fedc092b874361b91f71701d089bbd (127.5.83.2:46461)
I20260808 04:07:00.774335 24187 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.2@KRBTEST.COM'} at 127.5.83.2:48213
I20260808 04:07:01.127195 24490 heartbeater.cc:464] Registering TS with master...
I20260808 04:07:01.127650 24490 heartbeater.cc:511] Master 127.5.83.62:43283 requested a full tablet report, sending...
I20260808 04:07:01.129108 24187 ts_manager.cc:194] Registered new tserver with Master: e743b97eff7f41cabd30b29116a1be2c (127.5.83.1:45467)
I20260808 04:07:01.130254 24187 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.1@KRBTEST.COM'} at 127.5.83.1:57987
W20260808 04:07:01.154079 24726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:07:01.154345 24726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:07:01.154516 24726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:07:01.163341 24726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:07:01.163458 24726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:07:01.163609 24726 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.3
I20260808 04:07:01.177238 24726 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:35565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.5.83.20:59983/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.3.keytab
--principal=kudu/127.5.83.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.3:33939
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.83.3
--webserver_port=44583
--tserver_master_addrs=127.5.83.62:43283
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.3
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:07:01.179056 24726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:07:01.180823 24726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Aug 08 04:07:01 dist-test-slave-1l3l krb5kdc[22611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162020, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.62@KRBTEST.COM for HTTP/127.5.83.20@KRBTEST.COM
W20260808 04:07:01.204002 24734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:07:01.204907 24735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:07:01.211591 24738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:07:01.215807 24726 server_base.cc:1034] running on GCE node
I20260808 04:07:01.216620 24726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:07:01.223994 24726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:07:01.231496 24726 hybrid_clock.cc:648] HybridClock initialized: now 1786162021231398 us; error 66 us; skew 500 ppm
I20260808 04:07:01.231933 24726 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:07:01 dist-test-slave-1l3l krb5kdc[22611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162021, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:07:01.235998 24726 init.cc:377] Logged in from keytab as kudu/127.5.83.3@KRBTEST.COM (short username kudu)
I20260808 04:07:01.239017 24726 webserver.cc:460] Webserver started at http://127.5.83.3:44583/ using document root <none> and password file <none>
I20260808 04:07:01.240022 24726 fs_manager.cc:359] Metadata directory not provided
I20260808 04:07:01.240226 24726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 08 04:07:01 dist-test-slave-1l3l krb5kdc[22611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162021, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.3@KRBTEST.COM for HTTP/127.5.83.20@KRBTEST.COM
I20260808 04:07:01.296725 24726 fs_manager.cc:709] Time spent opening directory manager: real 0.020s user 0.008s sys 0.006s
I20260808 04:07:01.312505 24750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:07:01.314628 24726 fs_manager.cc:725] Time spent opening block manager: real 0.016s user 0.011s sys 0.004s
I20260808 04:07:01.314836 24726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-2/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-2/wal
uuid: "f64e5a37850f468b94f506a69e9baa00"
format_stamp: "Formatted at 2026-08-08 04:06:58 on dist-test-slave-1l3l"
server_key: "17c4b602b1316b8410f8b9216b5ddd6b"
server_key_iv: "d056a41fef3096fd4fb6f7c6c957875f"
server_key_version: "kuduclusterkey@0"
I20260808 04:07:01.316277 24726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:07:01.344889 24726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:07:01.349808 24726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:07:01.350104 24726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:07:01.351286 24726 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:07:01.352963 24726 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:07:01.353071 24726 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:07:01.353174 24726 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:07:01.353245 24726 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:07:01.391598 24726 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.3:33939
I20260808 04:07:01.391681 24915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.3:33939 every 8 connection(s)
I20260808 04:07:01.393632 24726 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
I20260808 04:07:01.403780 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 24726
Aug 08 04:07:01 dist-test-slave-1l3l krb5kdc[22611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162021, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.3@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:07:01.433295 24925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:01.396456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.3:49753 (local address 127.5.83.62:43283)
0808 04:07:01.415222 (+ 18766us) server_negotiation.cc:206] Beginning negotiation
0808 04:07:01.415228 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:07:01.415249 (+ 21us) server_negotiation.cc:407] Connection header received
0808 04:07:01.415333 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:07:01.415338 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:07:01.415430 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:07:01.415572 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:07:01.417602 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:01.418750 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:01.421082 (+ 2332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:01.421653 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:01.421740 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:01.426102 (+ 4362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:07:01.426110 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:07:01.426115 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:07:01.426152 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:07:01.429704 (+ 3552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:01.430644 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:01.430650 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:01.430654 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:01.430797 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:01.431340 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:01.431345 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:01.431348 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:01.431679 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:07:01.431851 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0808 04:07:01.432679 (+ 828us) server_negotiation.cc:299] Negotiation successful
0808 04:07:01.432896 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18644,"thread_start_us":23368,"threads_started":1}
I20260808 04:07:01.434338 24922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:01.397159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:43283 (local address 127.5.83.3:49753)
0808 04:07:01.404225 (+ 7066us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:01.404277 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 04:07:01.407963 (+ 3686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:07:01.416017 (+ 8054us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:07:01.416028 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:07:01.416609 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:07:01.417394 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:07:01.417418 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:07:01.418950 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:07:01.418954 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:07:01.420906 (+ 1952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:07:01.420942 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:07:01.422085 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:07:01.422092 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:07:01.422401 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:01.423138 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:07:01.423168 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:07:01.425831 (+ 2663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:07:01.429858 (+ 4027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:07:01.429868 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:07:01.429885 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:07:01.430490 (+ 605us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:07:01.430918 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:07:01.430924 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:07:01.430928 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:07:01.431191 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:07:01.431854 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:07:01.431862 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:07:01.432255 (+ 393us) client_negotiation.cc:769] Sending connection context
0808 04:07:01.432537 (+ 282us) client_negotiation.cc:240] Negotiation successful
0808 04:07:01.433042 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6764,"thread_start_us":14324,"threads_started":1}
I20260808 04:07:01.435791 24917 heartbeater.cc:347] Connected to a master server at 127.5.83.62:43283
I20260808 04:07:01.436142 24917 heartbeater.cc:464] Registering TS with master...
I20260808 04:07:01.437028 24917 heartbeater.cc:511] Master 127.5.83.62:43283 requested a full tablet report, sending...
I20260808 04:07:01.442975 24187 ts_manager.cc:194] Registered new tserver with Master: f64e5a37850f468b94f506a69e9baa00 (127.5.83.3:33939)
I20260808 04:07:01.444056 24187 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.3@KRBTEST.COM'} at 127.5.83.3:49753
I20260808 04:07:01.446426 24921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:01.404848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59656 (local address 127.5.83.62:43283)
0808 04:07:01.420098 (+ 15250us) server_negotiation.cc:206] Beginning negotiation
0808 04:07:01.420104 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:07:01.420122 (+ 18us) server_negotiation.cc:407] Connection header received
0808 04:07:01.420166 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:07:01.420171 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:07:01.420242 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:07:01.420348 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:07:01.422532 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:01.423966 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:01.425587 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:01.426230 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:01.426316 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:01.438321 (+ 12005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:07:01.438329 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:07:01.438334 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:07:01.438390 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:07:01.441725 (+ 3335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:01.442501 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:01.442507 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:01.442510 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:01.442660 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:01.444782 (+ 2122us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:01.444787 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:01.444790 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:01.445134 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:07:01.445386 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0808 04:07:01.445850 (+ 464us) server_negotiation.cc:299] Negotiation successful
0808 04:07:01.446071 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15127,"thread_start_us":10147,"threads_started":1}
I20260808 04:07:01.449302 5452 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:07:01.470321 24921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:01.452901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59664 (local address 127.5.83.62:43283)
0808 04:07:01.453056 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0808 04:07:01.453061 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:07:01.453576 (+ 515us) server_negotiation.cc:407] Connection header received
0808 04:07:01.453747 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:07:01.453752 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:07:01.453841 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:07:01.453932 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:07:01.455994 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:01.457016 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:01.458995 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:01.459633 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:01.459770 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:01.463110 (+ 3340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:07:01.463117 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:07:01.463122 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:07:01.463160 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:07:01.466644 (+ 3484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:01.467521 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:01.467527 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:01.467531 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:01.467676 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:01.468170 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:01.468176 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:01.468179 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:01.468544 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:07:01.468791 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0808 04:07:01.469728 (+ 937us) server_negotiation.cc:299] Negotiation successful
0808 04:07:01.469982 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260808 04:07:01.476241 24187 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59664:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:07:01.486397 24187 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
Aug 08 04:07:01 dist-test-slave-1l3l krb5kdc[22611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162020, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.62@KRBTEST.COM for HTTP/127.5.83.20@KRBTEST.COM
I20260808 04:07:01.537986 24940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:01.507917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:45467 (local address 127.0.0.1:53306)
0808 04:07:01.517296 (+ 9379us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:01.517313 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:07:01.517435 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:07:01.529847 (+ 12412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:07:01.529854 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:07:01.529898 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:07:01.530312 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:07:01.530324 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:07:01.532384 (+ 2060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:07:01.532389 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:07:01.534704 (+ 2315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:07:01.534719 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:07:01.536899 (+ 2180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:07:01.536906 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:07:01.537142 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:01.537192 (+ 50us) client_negotiation.cc:769] Sending connection context
0808 04:07:01.537323 (+ 131us) client_negotiation.cc:240] Negotiation successful
0808 04:07:01.537547 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9264,"mutex_wait_us":54,"thread_start_us":7711,"threads_started":1}
I20260808 04:07:01.538026 24941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:01.507683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43920 (local address 127.5.83.2:46461)
0808 04:07:01.528942 (+ 21259us) server_negotiation.cc:206] Beginning negotiation
0808 04:07:01.528951 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 04:07:01.528977 (+ 26us) server_negotiation.cc:407] Connection header received
0808 04:07:01.529066 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:07:01.529073 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:07:01.529302 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:07:01.529490 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:07:01.531219 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:01.532753 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:01.535251 (+ 2498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:01.536673 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:01.536792 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:01.536971 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0808 04:07:01.537571 (+ 600us) server_negotiation.cc:299] Negotiation successful
0808 04:07:01.537734 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21085,"thread_start_us":20963,"threads_started":1}
I20260808 04:07:01.539534 24943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:01.508490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53306 (local address 127.5.83.1:45467)
0808 04:07:01.516884 (+ 8394us) server_negotiation.cc:206] Beginning negotiation
0808 04:07:01.516894 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 04:07:01.527454 (+ 10560us) server_negotiation.cc:407] Connection header received
0808 04:07:01.527570 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:07:01.527579 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:07:01.527886 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:07:01.528077 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:07:01.530497 (+ 2420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:01.532185 (+ 1688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:01.534880 (+ 2695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:01.536346 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:01.536453 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:01.536703 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0808 04:07:01.539052 (+ 2349us) server_negotiation.cc:299] Negotiation successful
0808 04:07:01.539218 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8218,"thread_start_us":8089,"threads_started":1}
I20260808 04:07:01.538681 24942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:01.507186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.2:46461 (local address 127.0.0.1:43920)
0808 04:07:01.516069 (+ 8883us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:01.516117 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 04:07:01.516305 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:07:01.530531 (+ 14226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:07:01.530535 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:07:01.530561 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:07:01.530889 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:07:01.530902 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:07:01.532935 (+ 2033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:07:01.532940 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:07:01.535045 (+ 2105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:07:01.535061 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:07:01.537144 (+ 2083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:07:01.537151 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:07:01.537321 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:01.537366 (+ 45us) client_negotiation.cc:769] Sending connection context
0808 04:07:01.537453 (+ 87us) client_negotiation.cc:240] Negotiation successful
0808 04:07:01.538289 (+ 836us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8603,"mutex_wait_us":689,"thread_start_us":9855,"threads_started":1}
I20260808 04:07:01.543869 24399 tablet_service.cc:1467] Processing CreateTablet for tablet a76750c5adb64fda837c20fefca020b8 (DEFAULT_TABLE table=test-table [id=70a2e1ef054b4a3eb5a2556d374f7d0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:07:01.543876 24617 tablet_service.cc:1467] Processing CreateTablet for tablet a76750c5adb64fda837c20fefca020b8 (DEFAULT_TABLE table=test-table [id=70a2e1ef054b4a3eb5a2556d374f7d0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:07:01.545320 24399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a76750c5adb64fda837c20fefca020b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:07:01.545308 24617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a76750c5adb64fda837c20fefca020b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:07:01.545871 24944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:01.517671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.3:33939 (local address 127.0.0.1:51084)
0808 04:07:01.518190 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:01.518212 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 04:07:01.518317 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:07:01.538940 (+ 20623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:07:01.538944 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:07:01.538969 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:07:01.539254 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:07:01.539265 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:07:01.541447 (+ 2182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:07:01.541452 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:07:01.543362 (+ 1910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:07:01.543376 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:07:01.545155 (+ 1779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:07:01.545162 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:07:01.545333 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:01.545365 (+ 32us) client_negotiation.cc:769] Sending connection context
0808 04:07:01.545435 (+ 70us) client_negotiation.cc:240] Negotiation successful
0808 04:07:01.545554 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":275,"threads_started":1}
I20260808 04:07:01.547108 24946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:01.517783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51084 (local address 127.5.83.3:33939)
0808 04:07:01.538200 (+ 20417us) server_negotiation.cc:206] Beginning negotiation
0808 04:07:01.538208 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:07:01.538237 (+ 29us) server_negotiation.cc:407] Connection header received
0808 04:07:01.538323 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:07:01.538329 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:07:01.538535 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:07:01.539432 (+ 897us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:07:01.539760 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:01.541294 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:01.543556 (+ 2262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:01.544920 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:01.546350 (+ 1430us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:01.546558 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0808 04:07:01.546625 (+ 67us) server_negotiation.cc:299] Negotiation successful
0808 04:07:01.546768 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20238,"thread_start_us":20888,"threads_started":1}
I20260808 04:07:01.550601 24821 tablet_service.cc:1467] Processing CreateTablet for tablet a76750c5adb64fda837c20fefca020b8 (DEFAULT_TABLE table=test-table [id=70a2e1ef054b4a3eb5a2556d374f7d0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:07:01.552066 24821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a76750c5adb64fda837c20fefca020b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:07:01.597201 24947 tablet_bootstrap.cc:492] T a76750c5adb64fda837c20fefca020b8 P 64fedc092b874361b91f71701d089bbd: Bootstrap starting.
I20260808 04:07:01.598079 24948 tablet_bootstrap.cc:492] T a76750c5adb64fda837c20fefca020b8 P e743b97eff7f41cabd30b29116a1be2c: Bootstrap starting.
I20260808 04:07:01.599329 24949 tablet_bootstrap.cc:492] T a76750c5adb64fda837c20fefca020b8 P f64e5a37850f468b94f506a69e9baa00: Bootstrap starting.
I20260808 04:07:01.603000 24949 tablet_bootstrap.cc:654] T a76750c5adb64fda837c20fefca020b8 P f64e5a37850f468b94f506a69e9baa00: Neither blocks nor log segments found. Creating new log.
I20260808 04:07:01.604431 24949 log.cc:824] T a76750c5adb64fda837c20fefca020b8 P f64e5a37850f468b94f506a69e9baa00: Log is configured to *not* fsync() on all Append() calls
I20260808 04:07:01.609122 24947 tablet_bootstrap.cc:654] T a76750c5adb64fda837c20fefca020b8 P 64fedc092b874361b91f71701d089bbd: Neither blocks nor log segments found. Creating new log.
I20260808 04:07:01.611732 24948 tablet_bootstrap.cc:654] T a76750c5adb64fda837c20fefca020b8 P e743b97eff7f41cabd30b29116a1be2c: Neither blocks nor log segments found. Creating new log.
I20260808 04:07:01.613117 24948 log.cc:824] T a76750c5adb64fda837c20fefca020b8 P e743b97eff7f41cabd30b29116a1be2c: Log is configured to *not* fsync() on all Append() calls
I20260808 04:07:01.614033 24949 tablet_bootstrap.cc:492] T a76750c5adb64fda837c20fefca020b8 P f64e5a37850f468b94f506a69e9baa00: No bootstrap required, opened a new log
I20260808 04:07:01.614372 24949 ts_tablet_manager.cc:1397] T a76750c5adb64fda837c20fefca020b8 P f64e5a37850f468b94f506a69e9baa00: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.010s
I20260808 04:07:01.614470 24947 log.cc:824] T a76750c5adb64fda837c20fefca020b8 P 64fedc092b874361b91f71701d089bbd: Log is configured to *not* fsync() on all Append() calls
I20260808 04:07:01.624737 24948 tablet_bootstrap.cc:492] T a76750c5adb64fda837c20fefca020b8 P e743b97eff7f41cabd30b29116a1be2c: No bootstrap required, opened a new log
I20260808 04:07:01.625172 24948 ts_tablet_manager.cc:1397] T a76750c5adb64fda837c20fefca020b8 P e743b97eff7f41cabd30b29116a1be2c: Time spent bootstrapping tablet: real 0.027s user 0.005s sys 0.006s
I20260808 04:07:01.625617 24949 raft_consensus.cc:348] T a76750c5adb64fda837c20fefca020b8 P f64e5a37850f468b94f506a69e9baa00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64fedc092b874361b91f71701d089bbd" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 46461 } } peers { permanent_uuid: "f64e5a37850f468b94f506a69e9baa00" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 33939 } } peers { permanent_uuid: "e743b97eff7f41cabd30b29116a1be2c" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 45467 } }
I20260808 04:07:01.626170 24949 raft_consensus.cc:374] T a76750c5adb64fda837c20fefca020b8 P f64e5a37850f468b94f506a69e9baa00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:07:01.626307 24949 raft_consensus.cc:729] T a76750c5adb64fda837c20fefca020b8 P f64e5a37850f468b94f506a69e9baa00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f64e5a37850f468b94f506a69e9baa00, State: Initialized, Role: FOLLOWER
I20260808 04:07:01.627164 24949 consensus_queue.cc:260] T a76750c5adb64fda837c20fefca020b8 P f64e5a37850f468b94f506a69e9baa00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64fedc092b874361b91f71701d089bbd" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 46461 } } peers { permanent_uuid: "f64e5a37850f468b94f506a69e9baa00" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 33939 } } peers { permanent_uuid: "e743b97eff7f41cabd30b29116a1be2c" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 45467 } }
I20260808 04:07:01.632105 24947 tablet_bootstrap.cc:492] T a76750c5adb64fda837c20fefca020b8 P 64fedc092b874361b91f71701d089bbd: No bootstrap required, opened a new log
I20260808 04:07:01.632462 24947 ts_tablet_manager.cc:1397] T a76750c5adb64fda837c20fefca020b8 P 64fedc092b874361b91f71701d089bbd: Time spent bootstrapping tablet: real 0.036s user 0.012s sys 0.000s
I20260808 04:07:01.636281 24949 ts_tablet_manager.cc:1428] T a76750c5adb64fda837c20fefca020b8 P f64e5a37850f468b94f506a69e9baa00: Time spent starting tablet: real 0.022s user 0.019s sys 0.000s
I20260808 04:07:01.636688 24917 heartbeater.cc:503] Master 127.5.83.62:43283 was elected leader, sending a full tablet report...
I20260808 04:07:01.645722 24947 raft_consensus.cc:348] T a76750c5adb64fda837c20fefca020b8 P 64fedc092b874361b91f71701d089bbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64fedc092b874361b91f71701d089bbd" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 46461 } } peers { permanent_uuid: "f64e5a37850f468b94f506a69e9baa00" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 33939 } } peers { permanent_uuid: "e743b97eff7f41cabd30b29116a1be2c" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 45467 } }
I20260808 04:07:01.650533 24948 raft_consensus.cc:348] T a76750c5adb64fda837c20fefca020b8 P e743b97eff7f41cabd30b29116a1be2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64fedc092b874361b91f71701d089bbd" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 46461 } } peers { permanent_uuid: "f64e5a37850f468b94f506a69e9baa00" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 33939 } } peers { permanent_uuid: "e743b97eff7f41cabd30b29116a1be2c" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 45467 } }
I20260808 04:07:01.651054 24947 raft_consensus.cc:374] T a76750c5adb64fda837c20fefca020b8 P 64fedc092b874361b91f71701d089bbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:07:01.651243 24948 raft_consensus.cc:374] T a76750c5adb64fda837c20fefca020b8 P e743b97eff7f41cabd30b29116a1be2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:07:01.651407 24947 raft_consensus.cc:729] T a76750c5adb64fda837c20fefca020b8 P 64fedc092b874361b91f71701d089bbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64fedc092b874361b91f71701d089bbd, State: Initialized, Role: FOLLOWER
I20260808 04:07:01.651584 24948 raft_consensus.cc:729] T a76750c5adb64fda837c20fefca020b8 P e743b97eff7f41cabd30b29116a1be2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e743b97eff7f41cabd30b29116a1be2c, State: Initialized, Role: FOLLOWER
W20260808 04:07:01.652194 24918 tablet.cc:2369] T a76750c5adb64fda837c20fefca020b8 P f64e5a37850f468b94f506a69e9baa00: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:07:01.653232 24955 raft_consensus.cc:482] T a76750c5adb64fda837c20fefca020b8 P f64e5a37850f468b94f506a69e9baa00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:07:01.653472 24955 raft_consensus.cc:504] T a76750c5adb64fda837c20fefca020b8 P f64e5a37850f468b94f506a69e9baa00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64fedc092b874361b91f71701d089bbd" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 46461 } } peers { permanent_uuid: "f64e5a37850f468b94f506a69e9baa00" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 33939 } } peers { permanent_uuid: "e743b97eff7f41cabd30b29116a1be2c" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 45467 } }
I20260808 04:07:01.654208 24948 consensus_queue.cc:260] T a76750c5adb64fda837c20fefca020b8 P e743b97eff7f41cabd30b29116a1be2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64fedc092b874361b91f71701d089bbd" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 46461 } } peers { permanent_uuid: "f64e5a37850f468b94f506a69e9baa00" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 33939 } } peers { permanent_uuid: "e743b97eff7f41cabd30b29116a1be2c" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 45467 } }
I20260808 04:07:01.654712 24947 consensus_queue.cc:260] T a76750c5adb64fda837c20fefca020b8 P 64fedc092b874361b91f71701d089bbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64fedc092b874361b91f71701d089bbd" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 46461 } } peers { permanent_uuid: "f64e5a37850f468b94f506a69e9baa00" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 33939 } } peers { permanent_uuid: "e743b97eff7f41cabd30b29116a1be2c" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 45467 } }
I20260808 04:07:01.655367 24955 leader_election.cc:288] T a76750c5adb64fda837c20fefca020b8 P f64e5a37850f468b94f506a69e9baa00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64fedc092b874361b91f71701d089bbd (127.5.83.2:46461), e743b97eff7f41cabd30b29116a1be2c (127.5.83.1:45467)
I20260808 04:07:01.665076 24922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:01.655917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:45467 (local address 127.5.83.3:44291)
0808 04:07:01.656081 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:01.656102 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 04:07:01.656220 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:07:01.656908 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:07:01.656914 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:07:01.656941 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:07:01.657118 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:07:01.657128 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:07:01.660288 (+ 3160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:07:01.660294 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:07:01.662240 (+ 1946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:07:01.662256 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:07:01.664342 (+ 2086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:07:01.664349 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:07:01.664528 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:01.664563 (+ 35us) client_negotiation.cc:769] Sending connection context
0808 04:07:01.664657 (+ 94us) client_negotiation.cc:240] Negotiation successful
0808 04:07:01.664788 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260808 04:07:01.675933 24941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:01.656041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.3:33241 (local address 127.5.83.2:46461)
0808 04:07:01.656800 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0808 04:07:01.656806 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:07:01.666254 (+ 9448us) server_negotiation.cc:407] Connection header received
0808 04:07:01.666817 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:07:01.666822 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:07:01.666913 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:07:01.666975 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:07:01.667948 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:01.668974 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:01.673118 (+ 4144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:01.674321 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:01.674428 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:01.674522 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0808 04:07:01.675525 (+ 1003us) server_negotiation.cc:299] Negotiation successful
0808 04:07:01.675653 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626}
I20260808 04:07:01.676772 24922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:01.658493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.2:46461 (local address 127.5.83.3:33241)
0808 04:07:01.666172 (+ 7679us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:01.666190 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 04:07:01.666508 (+ 318us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:07:01.667185 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:07:01.667189 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:07:01.667215 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:07:01.667395 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:07:01.667407 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:07:01.669182 (+ 1775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:07:01.669186 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:07:01.670965 (+ 1779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:07:01.670981 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:07:01.674704 (+ 3723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:07:01.674710 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:07:01.674903 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:01.674935 (+ 32us) client_negotiation.cc:769] Sending connection context
0808 04:07:01.676354 (+ 1419us) client_negotiation.cc:240] Negotiation successful
0808 04:07:01.676498 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7583,"threads_started":1}
I20260808 04:07:01.665820 24943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:01.656034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.3:44291 (local address 127.5.83.1:45467)
0808 04:07:01.656263 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0808 04:07:01.656269 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:07:01.656289 (+ 20us) server_negotiation.cc:407] Connection header received
0808 04:07:01.656368 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:07:01.656373 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:07:01.656457 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:07:01.656548 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:07:01.657278 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:01.658344 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:01.662564 (+ 4220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:01.663996 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:01.664131 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:01.664237 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0808 04:07:01.665405 (+ 1168us) server_negotiation.cc:299] Negotiation successful
0808 04:07:01.665537 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260808 04:07:01.680629 24421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a76750c5adb64fda837c20fefca020b8" candidate_uuid: "f64e5a37850f468b94f506a69e9baa00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e743b97eff7f41cabd30b29116a1be2c" is_pre_election: true
I20260808 04:07:01.681224 24421 raft_consensus.cc:2454] T a76750c5adb64fda837c20fefca020b8 P e743b97eff7f41cabd30b29116a1be2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f64e5a37850f468b94f506a69e9baa00 in term 0.
I20260808 04:07:01.683199 24754 leader_election.cc:302] T a76750c5adb64fda837c20fefca020b8 P f64e5a37850f468b94f506a69e9baa00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e743b97eff7f41cabd30b29116a1be2c, f64e5a37850f468b94f506a69e9baa00; no voters:
I20260808 04:07:01.683665 24955 raft_consensus.cc:2781] T a76750c5adb64fda837c20fefca020b8 P f64e5a37850f468b94f506a69e9baa00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:07:01.683840 24955 raft_consensus.cc:482] T a76750c5adb64fda837c20fefca020b8 P f64e5a37850f468b94f506a69e9baa00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:07:01.683943 24955 raft_consensus.cc:3037] T a76750c5adb64fda837c20fefca020b8 P f64e5a37850f468b94f506a69e9baa00 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:07:01.686986 24955 raft_consensus.cc:504] T a76750c5adb64fda837c20fefca020b8 P f64e5a37850f468b94f506a69e9baa00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64fedc092b874361b91f71701d089bbd" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 46461 } } peers { permanent_uuid: "f64e5a37850f468b94f506a69e9baa00" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 33939 } } peers { permanent_uuid: "e743b97eff7f41cabd30b29116a1be2c" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 45467 } }
I20260808 04:07:01.688150 24421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a76750c5adb64fda837c20fefca020b8" candidate_uuid: "f64e5a37850f468b94f506a69e9baa00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e743b97eff7f41cabd30b29116a1be2c"
I20260808 04:07:01.688349 24421 raft_consensus.cc:3037] T a76750c5adb64fda837c20fefca020b8 P e743b97eff7f41cabd30b29116a1be2c [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:07:01.691334 24421 raft_consensus.cc:2454] T a76750c5adb64fda837c20fefca020b8 P e743b97eff7f41cabd30b29116a1be2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f64e5a37850f468b94f506a69e9baa00 in term 1.
I20260808 04:07:01.691756 24754 leader_election.cc:302] T a76750c5adb64fda837c20fefca020b8 P f64e5a37850f468b94f506a69e9baa00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e743b97eff7f41cabd30b29116a1be2c, f64e5a37850f468b94f506a69e9baa00; no voters:
I20260808 04:07:01.692143 24955 leader_election.cc:288] T a76750c5adb64fda837c20fefca020b8 P f64e5a37850f468b94f506a69e9baa00 [CANDIDATE]: Term 1 election: Requested vote from peers 64fedc092b874361b91f71701d089bbd (127.5.83.2:46461), e743b97eff7f41cabd30b29116a1be2c (127.5.83.1:45467)
I20260808 04:07:01.692309 24955 raft_consensus.cc:2781] T a76750c5adb64fda837c20fefca020b8 P f64e5a37850f468b94f506a69e9baa00 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:07:01.692433 24955 raft_consensus.cc:686] T a76750c5adb64fda837c20fefca020b8 P f64e5a37850f468b94f506a69e9baa00 [term 1 LEADER]: Becoming Leader. State: Replica: f64e5a37850f468b94f506a69e9baa00, State: Running, Role: LEADER
I20260808 04:07:01.693078 24955 consensus_queue.cc:237] T a76750c5adb64fda837c20fefca020b8 P f64e5a37850f468b94f506a69e9baa00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64fedc092b874361b91f71701d089bbd" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 46461 } } peers { permanent_uuid: "f64e5a37850f468b94f506a69e9baa00" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 33939 } } peers { permanent_uuid: "e743b97eff7f41cabd30b29116a1be2c" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 45467 } }
I20260808 04:07:01.695067 24644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a76750c5adb64fda837c20fefca020b8" candidate_uuid: "f64e5a37850f468b94f506a69e9baa00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64fedc092b874361b91f71701d089bbd" is_pre_election: true
I20260808 04:07:01.695696 24644 raft_consensus.cc:2454] T a76750c5adb64fda837c20fefca020b8 P 64fedc092b874361b91f71701d089bbd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f64e5a37850f468b94f506a69e9baa00 in term 0.
I20260808 04:07:01.698235 24644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a76750c5adb64fda837c20fefca020b8" candidate_uuid: "f64e5a37850f468b94f506a69e9baa00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64fedc092b874361b91f71701d089bbd"
I20260808 04:07:01.698436 24644 raft_consensus.cc:3037] T a76750c5adb64fda837c20fefca020b8 P 64fedc092b874361b91f71701d089bbd [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:07:01.701639 24644 raft_consensus.cc:2454] T a76750c5adb64fda837c20fefca020b8 P 64fedc092b874361b91f71701d089bbd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f64e5a37850f468b94f506a69e9baa00 in term 1.
I20260808 04:07:01.713617 24187 catalog_manager.cc:5495] T a76750c5adb64fda837c20fefca020b8 P f64e5a37850f468b94f506a69e9baa00 reported cstate change: term changed from 0 to 1, leader changed from <none> to f64e5a37850f468b94f506a69e9baa00 (127.5.83.3). New cstate: current_term: 1 leader_uuid: "f64e5a37850f468b94f506a69e9baa00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64fedc092b874361b91f71701d089bbd" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 46461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f64e5a37850f468b94f506a69e9baa00" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 33939 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e743b97eff7f41cabd30b29116a1be2c" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 45467 } health_report { overall_health: UNKNOWN } } }
I20260808 04:07:01.720278 24948 ts_tablet_manager.cc:1428] T a76750c5adb64fda837c20fefca020b8 P e743b97eff7f41cabd30b29116a1be2c: Time spent starting tablet: real 0.095s user 0.005s sys 0.018s
I20260808 04:07:01.721207 24716 heartbeater.cc:503] Master 127.5.83.62:43283 was elected leader, sending a full tablet report...
I20260808 04:07:01.722273 24947 ts_tablet_manager.cc:1428] T a76750c5adb64fda837c20fefca020b8 P 64fedc092b874361b91f71701d089bbd: Time spent starting tablet: real 0.090s user 0.009s sys 0.014s
I20260808 04:07:01.722934 24490 heartbeater.cc:503] Master 127.5.83.62:43283 was elected leader, sending a full tablet report...
W20260808 04:07:01.728830 24717 tablet.cc:2369] T a76750c5adb64fda837c20fefca020b8 P 64fedc092b874361b91f71701d089bbd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:07:01.769275 24946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:01.761725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51094 (local address 127.5.83.3:33939)
0808 04:07:01.761926 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0808 04:07:01.761930 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:07:01.761946 (+ 16us) server_negotiation.cc:407] Connection header received
0808 04:07:01.762030 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:07:01.762034 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:07:01.762108 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:07:01.762247 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:07:01.762714 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:01.763900 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:01.766115 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:01.766742 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:01.766870 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:01.767555 (+ 685us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:07:01.768250 (+ 695us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:07:01.768386 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0808 04:07:01.768917 (+ 531us) server_negotiation.cc:299] Negotiation successful
0808 04:07:01.769001 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 04:07:01.786724 24421 raft_consensus.cc:1261] T a76750c5adb64fda837c20fefca020b8 P e743b97eff7f41cabd30b29116a1be2c [term 1 FOLLOWER]: Refusing update from remote peer f64e5a37850f468b94f506a69e9baa00: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:07:01.786780 24644 raft_consensus.cc:1261] T a76750c5adb64fda837c20fefca020b8 P 64fedc092b874361b91f71701d089bbd [term 1 FOLLOWER]: Refusing update from remote peer f64e5a37850f468b94f506a69e9baa00: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:07:01.787789 24963 consensus_queue.cc:1035] T a76750c5adb64fda837c20fefca020b8 P f64e5a37850f468b94f506a69e9baa00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e743b97eff7f41cabd30b29116a1be2c" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 45467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:07:01.788694 24963 consensus_queue.cc:1035] T a76750c5adb64fda837c20fefca020b8 P f64e5a37850f468b94f506a69e9baa00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64fedc092b874361b91f71701d089bbd" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 46461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:07:01.811839 24967 mvcc.cc:205] Tried to move back new op lower bound from 7316119641227350016 to 7316119640859213824. Current Snapshot: MvccSnapshot[applied={T|T < 7316119641227350016}]
I20260808 04:07:01.817757 24966 mvcc.cc:205] Tried to move back new op lower bound from 7316119641227350016 to 7316119640859213824. Current Snapshot: MvccSnapshot[applied={T|T < 7316119641227350016 or (T in {7316119641227350016})}]
W20260808 04:07:01.818270 24492 tablet.cc:2369] T a76750c5adb64fda837c20fefca020b8 P e743b97eff7f41cabd30b29116a1be2c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:07:01.830530 24187 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:07:01.830883 24187 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:07:01.835086 24187 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4ce7014684154846afcdf2e7a618d8b5: Sending DeleteTablet for 3 replicas of tablet a76750c5adb64fda837c20fefca020b8
I20260808 04:07:01.836228 24617 tablet_service.cc:1514] Processing DeleteTablet for tablet a76750c5adb64fda837c20fefca020b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:07:01 UTC) from {username='kudu', principal='kudu/127.5.83.62@KRBTEST.COM'} at 127.0.0.1:43920
I20260808 04:07:01.840356 24821 tablet_service.cc:1514] Processing DeleteTablet for tablet a76750c5adb64fda837c20fefca020b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:07:01 UTC) from {username='kudu', principal='kudu/127.5.83.62@KRBTEST.COM'} at 127.0.0.1:51084
I20260808 04:07:01.841244 24980 tablet_replica.cc:330] T a76750c5adb64fda837c20fefca020b8 P f64e5a37850f468b94f506a69e9baa00: stopping tablet replica
I20260808 04:07:01.841683 24980 raft_consensus.cc:2229] T a76750c5adb64fda837c20fefca020b8 P f64e5a37850f468b94f506a69e9baa00 [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:07:01.842027 24399 tablet_service.cc:1514] Processing DeleteTablet for tablet a76750c5adb64fda837c20fefca020b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:07:01 UTC) from {username='kudu', principal='kudu/127.5.83.62@KRBTEST.COM'} at 127.0.0.1:53306
I20260808 04:07:01.842280 24980 raft_consensus.cc:2258] T a76750c5adb64fda837c20fefca020b8 P f64e5a37850f468b94f506a69e9baa00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:07:01.844769 24979 tablet_replica.cc:330] T a76750c5adb64fda837c20fefca020b8 P 64fedc092b874361b91f71701d089bbd: stopping tablet replica
I20260808 04:07:01.845165 24979 raft_consensus.cc:2229] T a76750c5adb64fda837c20fefca020b8 P 64fedc092b874361b91f71701d089bbd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:07:01.845616 24979 raft_consensus.cc:2258] T a76750c5adb64fda837c20fefca020b8 P 64fedc092b874361b91f71701d089bbd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:07:01.846312 24981 tablet_replica.cc:330] T a76750c5adb64fda837c20fefca020b8 P e743b97eff7f41cabd30b29116a1be2c: stopping tablet replica
I20260808 04:07:01.847786 24979 ts_tablet_manager.cc:1905] T a76750c5adb64fda837c20fefca020b8 P 64fedc092b874361b91f71701d089bbd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:07:01.850394 24981 raft_consensus.cc:2229] T a76750c5adb64fda837c20fefca020b8 P e743b97eff7f41cabd30b29116a1be2c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:07:01.851485 24981 raft_consensus.cc:2258] T a76750c5adb64fda837c20fefca020b8 P e743b97eff7f41cabd30b29116a1be2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:07:01.853504 24981 ts_tablet_manager.cc:1905] T a76750c5adb64fda837c20fefca020b8 P e743b97eff7f41cabd30b29116a1be2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:07:01.853694 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 24248
I20260808 04:07:01.854422 24979 ts_tablet_manager.cc:1918] T a76750c5adb64fda837c20fefca020b8 P 64fedc092b874361b91f71701d089bbd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:07:01.854560 24979 log.cc:1196] T a76750c5adb64fda837c20fefca020b8 P 64fedc092b874361b91f71701d089bbd: Deleting WAL directory at /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-1/wal/wals/a76750c5adb64fda837c20fefca020b8
I20260808 04:07:01.855101 24979 ts_tablet_manager.cc:1939] T a76750c5adb64fda837c20fefca020b8 P 64fedc092b874361b91f71701d089bbd: Deleting consensus metadata
I20260808 04:07:01.857062 24164 catalog_manager.cc:4841] TS 64fedc092b874361b91f71701d089bbd (127.5.83.2:46461): tablet a76750c5adb64fda837c20fefca020b8 (table test-table [id=70a2e1ef054b4a3eb5a2556d374f7d0c]) successfully deleted
I20260808 04:07:01.862732 24981 ts_tablet_manager.cc:1918] T a76750c5adb64fda837c20fefca020b8 P e743b97eff7f41cabd30b29116a1be2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:07:01.862890 24981 log.cc:1196] T a76750c5adb64fda837c20fefca020b8 P e743b97eff7f41cabd30b29116a1be2c: Deleting WAL directory at /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-0/wal/wals/a76750c5adb64fda837c20fefca020b8
I20260808 04:07:01.863430 24981 ts_tablet_manager.cc:1939] T a76750c5adb64fda837c20fefca020b8 P e743b97eff7f41cabd30b29116a1be2c: Deleting consensus metadata
I20260808 04:07:01.865140 24168 catalog_manager.cc:4841] TS e743b97eff7f41cabd30b29116a1be2c (127.5.83.1:45467): tablet a76750c5adb64fda837c20fefca020b8 (table test-table [id=70a2e1ef054b4a3eb5a2556d374f7d0c]) successfully deleted
I20260808 04:07:01.867852 24980 ts_tablet_manager.cc:1905] T a76750c5adb64fda837c20fefca020b8 P f64e5a37850f468b94f506a69e9baa00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:07:01.872447 24943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:01.854809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53322 (local address 127.5.83.1:45467)
0808 04:07:01.855101 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0808 04:07:01.855106 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:07:01.855128 (+ 22us) server_negotiation.cc:407] Connection header received
0808 04:07:01.855357 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:07:01.855362 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:07:01.855451 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:07:01.855564 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:07:01.857455 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:01.858651 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:01.860672 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:01.861361 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:01.861495 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:01.864805 (+ 3310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:07:01.864816 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:07:01.864834 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:07:01.864885 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:07:01.868536 (+ 3651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:01.869360 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:01.869369 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:01.869373 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:01.869527 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:01.869967 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:01.869972 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:01.869975 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:01.870395 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:07:01.870689 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0808 04:07:01.871823 (+ 1134us) server_negotiation.cc:299] Negotiation successful
0808 04:07:01.872059 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168}
I20260808 04:07:01.873116 24482 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:07:01.874629 24980 ts_tablet_manager.cc:1918] T a76750c5adb64fda837c20fefca020b8 P f64e5a37850f468b94f506a69e9baa00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:07:01.874804 24980 log.cc:1196] T a76750c5adb64fda837c20fefca020b8 P f64e5a37850f468b94f506a69e9baa00: Deleting WAL directory at /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161826605655-5452-0/minicluster-data/ts-2/wal/wals/a76750c5adb64fda837c20fefca020b8
I20260808 04:07:01.881048 24980 ts_tablet_manager.cc:1939] T a76750c5adb64fda837c20fefca020b8 P f64e5a37850f468b94f506a69e9baa00: Deleting consensus metadata
I20260808 04:07:01.882907 24164 catalog_manager.cc:4841] TS f64e5a37850f468b94f506a69e9baa00 (127.5.83.3:33939): tablet a76750c5adb64fda837c20fefca020b8 (table test-table [id=70a2e1ef054b4a3eb5a2556d374f7d0c]) successfully deleted
I20260808 04:07:02.027550 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 24248
I20260808 04:07:02.057471 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 24498
I20260808 04:07:02.075356 24941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:02.058405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43930 (local address 127.5.83.2:46461)
0808 04:07:02.058592 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0808 04:07:02.058599 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:07:02.058654 (+ 55us) server_negotiation.cc:407] Connection header received
0808 04:07:02.058858 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:07:02.058864 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:07:02.058968 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:07:02.059076 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:07:02.061009 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:02.062159 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:02.064124 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:02.064684 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:02.064812 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:02.068177 (+ 3365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:07:02.068196 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:07:02.068216 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:07:02.068257 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:07:02.072079 (+ 3822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:02.072955 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:02.072965 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:02.072969 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:02.073132 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:02.073545 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:02.073550 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:02.073553 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:02.074053 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:07:02.074354 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0808 04:07:02.074691 (+ 337us) server_negotiation.cc:299] Negotiation successful
0808 04:07:02.074956 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 04:07:02.076220 24707 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:07:02.227185 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 24498
I20260808 04:07:02.255798 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 24726
I20260808 04:07:02.272938 24946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:02.256675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51098 (local address 127.5.83.3:33939)
0808 04:07:02.256870 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0808 04:07:02.256876 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:07:02.257031 (+ 155us) server_negotiation.cc:407] Connection header received
0808 04:07:02.257235 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:07:02.257240 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:07:02.257338 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:07:02.257452 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:07:02.259304 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:02.260567 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:02.262413 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:02.263099 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:02.263250 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:02.265966 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:07:02.265986 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:07:02.266006 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:07:02.266062 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:07:02.269981 (+ 3919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:02.270663 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:02.270674 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:02.270681 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:02.270884 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:02.271256 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:02.271265 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:02.271268 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:02.271792 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:07:02.272118 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0808 04:07:02.272297 (+ 179us) server_negotiation.cc:299] Negotiation successful
0808 04:07:02.272521 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260808 04:07:02.273694 24908 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:07:02.395640 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 24726
I20260808 04:07:02.417605 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 24127
I20260808 04:07:02.418716 24235 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:07:02.527431 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 24127
I20260808 04:07:02.556650 5452 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260808 04:07:02.901055 5452 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260808 04:07:02.901279 5452 mini_ranger.cc:66] Stopping Ranger...
I20260808 04:07:03.245690 5452 mini_ranger.cc:68] Stopped Ranger
2026-08-08 04:07:03.245 UTC [22662] LOG: received smart shutdown request
2026-08-08 04:07:03.248 UTC [22662] LOG: background worker "logical replication launcher" (PID 22674) exited with exit code 1
2026-08-08 04:07:03.248 UTC [22669] LOG: shutting down
2026-08-08 04:07:03.376 UTC [22662] LOG: database system is shut down
2026-08-08T04:07:03Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (50889 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161826605655-5452-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:07:03 dist-test-slave-1l3l krb5kdc[25009](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:07:03 dist-test-slave-1l3l krb5kdc[25009](info): set up 2 sockets
Aug 08 04:07:03 dist-test-slave-1l3l krb5kdc[25009](info): commencing operation
krb5kdc: starting...
W20260808 04:07:04.659451 5452 mini_kdc.cc:121] Time spent starting KDC: real 1.137s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161826605655-5452-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161826605655-5452-0/krb5kdc/test-user.keytab.
Aug 08 04:07:04 dist-test-slave-1l3l krb5kdc[25009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:07:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:07:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.83.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab.
Entry for principal kudu/127.5.83.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab.
I20260808 04:07:04.883332 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161826605655-5452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161826605655-5452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161826605655-5452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:45997
--webserver_interface=127.5.83.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab
--principal=kudu/127.5.83.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:37863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.62:45997,127.5.83.61:35491,127.5.83.60:45383
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:07:05.202898 25027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:07:05.203177 25027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:07:05.203325 25027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:07:05.210435 25027 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:07:05.210531 25027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:07:05.210603 25027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:07:05.210685 25027 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:07:05.210741 25027 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:07:05.220163 25027 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:37863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161826605655-5452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161826605655-5452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.62:45997,127.5.83.61:35491,127.5.83.60:45383
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab
--principal=kudu/127.5.83.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:45997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161826605655-5452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:07:05.221385 25027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:07:05.222666 25027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:07:05.233955 25036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:07:05.234005 25038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:07:05.234406 25035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:07:05.234817 25027 server_base.cc:1034] running on GCE node
I20260808 04:07:05.235554 25027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:07:05.236800 25027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:07:05.237984 25027 hybrid_clock.cc:648] HybridClock initialized: now 1786162025237937 us; error 44 us; skew 500 ppm
I20260808 04:07:05.238286 25027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:07:05 dist-test-slave-1l3l krb5kdc[25009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162025, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:07:05.241206 25027 init.cc:377] Logged in from keytab as kudu/127.5.83.62@KRBTEST.COM (short username kudu)
I20260808 04:07:05.242497 25027 webserver.cc:460] Webserver started at http://127.5.83.62:34569/ using document root <none> and password file <none>
I20260808 04:07:05.243109 25027 fs_manager.cc:359] Metadata directory not provided
I20260808 04:07:05.243221 25027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:07:05.243501 25027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:07:05.246357 25027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161826605655-5452-0/minicluster-data/master-0/data/instance:
uuid: "7e7ccafc1d50430980f490ecdc271ca0"
format_stamp: "Formatted at 2026-08-08 04:07:05 on dist-test-slave-1l3l"
server_key: "70cd33e92684274aeb6ee93dcbdcd9f2"
server_key_iv: "cc482b2a16437a8ce1254ef8de8d7392"
server_key_version: "encryptionkey@0"
I20260808 04:07:05.247115 25027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161826605655-5452-0/minicluster-data/master-0/wal/instance:
uuid: "7e7ccafc1d50430980f490ecdc271ca0"
format_stamp: "Formatted at 2026-08-08 04:07:05 on dist-test-slave-1l3l"
server_key: "70cd33e92684274aeb6ee93dcbdcd9f2"
server_key_iv: "cc482b2a16437a8ce1254ef8de8d7392"
server_key_version: "encryptionkey@0"
I20260808 04:07:05.251902 25027 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260808 04:07:05.255110 25050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:07:05.256319 25027 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:07:05.256474 25027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161826605655-5452-0/minicluster-data/master-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161826605655-5452-0/minicluster-data/master-0/wal
uuid: "7e7ccafc1d50430980f490ecdc271ca0"
format_stamp: "Formatted at 2026-08-08 04:07:05 on dist-test-slave-1l3l"
server_key: "70cd33e92684274aeb6ee93dcbdcd9f2"
server_key_iv: "cc482b2a16437a8ce1254ef8de8d7392"
server_key_version: "encryptionkey@0"
I20260808 04:07:05.256633 25027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161826605655-5452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161826605655-5452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161826605655-5452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:07:05.272367 25027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:07:05.276330 25027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:07:05.276576 25027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:07:05.292100 25027 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.62:45997
I20260808 04:07:05.292140 25124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.62:45997 every 8 connection(s)
I20260808 04:07:05.293290 25027 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
I20260808 04:07:05.295697 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 25027
I20260808 04:07:05.295943 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161826605655-5452-0/minicluster-data/master-0/wal/instance
I20260808 04:07:05.296550 5452 external_mini_cluster.cc:1428] Setting key 5ae719c30cae0d60c144c317e1f6f3d8
I20260808 04:07:05.296890 25126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:07:05.302874 25126 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.83.62" port: 45997 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 04:07:05 dist-test-slave-1l3l krb5kdc[25009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162025, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.62@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.83.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.61@KRBTEST.COM" created.
I20260808 04:07:05.323764 25133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:05.305836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:45997 (local address 127.0.0.1:49702)
0808 04:07:05.306668 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:05.306713 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 04:07:05.308654 (+ 1941us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:07:05.309411 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:07:05.309417 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:07:05.310061 (+ 644us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:07:05.310380 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:07:05.310396 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:07:05.311923 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:07:05.311927 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:07:05.312959 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:07:05.312971 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:07:05.313833 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:07:05.313841 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:07:05.314047 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:05.314652 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:07:05.314672 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:07:05.316803 (+ 2131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:07:05.319805 (+ 3002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:07:05.319815 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:07:05.319826 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:07:05.320199 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:07:05.320573 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:07:05.320578 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:07:05.320582 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:07:05.320759 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:07:05.321505 (+ 746us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:07:05.321513 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:07:05.321723 (+ 210us) client_negotiation.cc:769] Sending connection context
0808 04:07:05.321970 (+ 247us) client_negotiation.cc:240] Negotiation successful
0808 04:07:05.322504 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"mutex_wait_us":130,"thread_start_us":383,"threads_started":1}
I20260808 04:07:05.323798 25134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:05.305847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49702 (local address 127.5.83.62:45997)
0808 04:07:05.307435 (+ 1588us) server_negotiation.cc:206] Beginning negotiation
0808 04:07:05.307444 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 04:07:05.307497 (+ 53us) server_negotiation.cc:407] Connection header received
0808 04:07:05.308989 (+ 1492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:07:05.308994 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:07:05.309196 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:07:05.309363 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:07:05.310560 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:05.311776 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:05.313086 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:05.313620 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:05.313775 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:05.317004 (+ 3229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:07:05.317012 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:07:05.317022 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:07:05.317050 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:07:05.319659 (+ 2609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:05.320314 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:05.320321 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:05.320323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:05.320481 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:05.320857 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:05.320861 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:05.320865 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:05.321377 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:07:05.321548 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0808 04:07:05.322094 (+ 546us) server_negotiation.cc:299] Negotiation successful
0808 04:07:05.322383 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":1385,"spinlock_wait_cycles":17152,"thread_start_us":1195,"threads_started":1}
I20260808 04:07:05.326557 25126 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.83.61" port: 35491 } has no permanent_uuid. Determining permanent_uuid...
I20260808 04:07:05.327634 25133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:05.327179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.61:35491 (local address 127.0.0.1:51560)
0808 04:07:05.327429 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:05.327532 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.61:35491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260808 04:07:05.327958 25055 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.83.61:35491: connect: Connection refused (error 111)
I20260808 04:07:05.329455 25133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:05.329128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.61:35491 (local address 127.0.0.1:51564)
0808 04:07:05.329254 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:05.329317 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.61:35491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260808 04:07:05.329650 25126 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.83.61:35491: Network error: Client connection negotiation failed: client connection to 127.5.83.61:35491: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.61.keytab.
Entry for principal kudu/127.5.83.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.61.keytab.
I20260808 04:07:05.348953 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161826605655-5452-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161826605655-5452-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161826605655-5452-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161826605655-5452-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.61:35491
--webserver_interface=127.5.83.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.61.keytab
--principal=kudu/127.5.83.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:37863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.62:45997,127.5.83.61:35491,127.5.83.60:45383
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:07:05.379087 25126 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.83.61" port: 35491 } attempt: 1
I20260808 04:07:05.380295 25133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:05.379816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.61:35491 (local address 127.0.0.1:51574)
0808 04:07:05.380012 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:05.380137 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.61:35491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260808 04:07:05.381274 25133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:05.380901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.61:35491 (local address 127.0.0.1:51588)
0808 04:07:05.381034 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:05.381114 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.61:35491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260808 04:07:05.381475 25126 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.83.61:35491: Network error: Client connection negotiation failed: client connection to 127.5.83.61:35491: connect: Connection refused (error 111)
I20260808 04:07:05.449736 25126 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.83.61" port: 35491 } attempt: 2
I20260808 04:07:05.451031 25133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:05.450459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.61:35491 (local address 127.0.0.1:51598)
0808 04:07:05.450719 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:05.450844 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.61:35491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260808 04:07:05.452216 25133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:05.451774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.61:35491 (local address 127.0.0.1:51604)
0808 04:07:05.451924 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:05.452033 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.61:35491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260808 04:07:05.452431 25126 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.83.61:35491: Network error: Client connection negotiation failed: client connection to 127.5.83.61:35491: connect: Connection refused (error 111)
I20260808 04:07:05.543696 25126 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.83.61" port: 35491 } attempt: 3
I20260808 04:07:05.545006 25133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:05.544443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.61:35491 (local address 127.0.0.1:51616)
0808 04:07:05.544679 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:05.544807 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.61:35491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260808 04:07:05.546140 25133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:05.545700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.61:35491 (local address 127.0.0.1:51630)
0808 04:07:05.545857 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:05.545961 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.61:35491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260808 04:07:05.546345 25126 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.83.61:35491: Network error: Client connection negotiation failed: client connection to 127.5.83.61:35491: connect: Connection refused (error 111)
W20260808 04:07:05.662706 25141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:07:05.662973 25141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:07:05.663061 25141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:07:05.670120 25141 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:07:05.670222 25141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:07:05.670266 25141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:07:05.670322 25141 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:07:05.670365 25141 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:07:05.679971 25141 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:37863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161826605655-5452-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161826605655-5452-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.62:45997,127.5.83.61:35491,127.5.83.60:45383
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.61.keytab
--principal=kudu/127.5.83.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.61:35491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161826605655-5452-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.5.83.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161826605655-5452-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:07:05.681195 25141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:07:05.682418 25141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 04:07:05.689589 25126 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.83.61" port: 35491 } attempt: 4
I20260808 04:07:05.690650 25133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:05.690139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.61:35491 (local address 127.0.0.1:51632)
0808 04:07:05.690435 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:05.690522 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.61:35491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
I20260808 04:07:05.691625 25133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:05.691230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.61:35491 (local address 127.0.0.1:51648)
0808 04:07:05.691365 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:05.691475 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.61:35491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260808 04:07:05.691887 25126 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.83.61:35491: Network error: Client connection negotiation failed: client connection to 127.5.83.61:35491: connect: Connection refused (error 111)
W20260808 04:07:05.693289 25147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:07:05.693499 25151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:07:05.693631 25148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:07:05.694325 25141 server_base.cc:1034] running on GCE node
I20260808 04:07:05.694965 25141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:07:05.696213 25141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:07:05.697417 25141 hybrid_clock.cc:648] HybridClock initialized: now 1786162025697351 us; error 43 us; skew 500 ppm
I20260808 04:07:05.697712 25141 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:07:05 dist-test-slave-1l3l krb5kdc[25009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162025, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:07:05.704646 25141 init.cc:377] Logged in from keytab as kudu/127.5.83.61@KRBTEST.COM (short username kudu)
I20260808 04:07:05.705940 25141 webserver.cc:460] Webserver started at http://127.5.83.61:41815/ using document root <none> and password file <none>
I20260808 04:07:05.706560 25141 fs_manager.cc:359] Metadata directory not provided
I20260808 04:07:05.706671 25141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:07:05.707017 25141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:07:05.709824 25141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161826605655-5452-0/minicluster-data/master-1/data/instance:
uuid: "2c936fbf17904ddfac4994949dc4065a"
format_stamp: "Formatted at 2026-08-08 04:07:05 on dist-test-slave-1l3l"
server_key: "6900b8f47d4541b37038e1164922c30f"
server_key_iv: "76fd579b0510ce0c02466a443746ddba"
server_key_version: "encryptionkey@0"
I20260808 04:07:05.710578 25141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161826605655-5452-0/minicluster-data/master-1/wal/instance:
uuid: "2c936fbf17904ddfac4994949dc4065a"
format_stamp: "Formatted at 2026-08-08 04:07:05 on dist-test-slave-1l3l"
server_key: "6900b8f47d4541b37038e1164922c30f"
server_key_iv: "76fd579b0510ce0c02466a443746ddba"
server_key_version: "encryptionkey@0"
I20260808 04:07:05.715209 25141 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260808 04:07:05.718613 25160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:07:05.719897 25141 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:07:05.720045 25141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161826605655-5452-0/minicluster-data/master-1/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161826605655-5452-0/minicluster-data/master-1/wal
uuid: "2c936fbf17904ddfac4994949dc4065a"
format_stamp: "Formatted at 2026-08-08 04:07:05 on dist-test-slave-1l3l"
server_key: "6900b8f47d4541b37038e1164922c30f"
server_key_iv: "76fd579b0510ce0c02466a443746ddba"
server_key_version: "encryptionkey@0"
I20260808 04:07:05.720214 25141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161826605655-5452-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161826605655-5452-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161826605655-5452-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:07:05.733604 25141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:07:05.737205 25141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:07:05.737463 25141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:07:05.754036 25141 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.61:35491
I20260808 04:07:05.754092 25237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.61:35491 every 8 connection(s)
I20260808 04:07:05.755414 25141 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161826605655-5452-0/minicluster-data/master-1/data/info.pb
I20260808 04:07:05.758599 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 25141
I20260808 04:07:05.758819 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161826605655-5452-0/minicluster-data/master-1/wal/instance
I20260808 04:07:05.759246 25238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:07:05.759423 5452 external_mini_cluster.cc:1428] Setting key 432a92de576f6b995a12cb3c6308e925
I20260808 04:07:05.767525 25238 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.83.62" port: 45997 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 04:07:05 dist-test-slave-1l3l krb5kdc[25009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162025, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.61@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.83.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.60@KRBTEST.COM" created.
I20260808 04:07:05.786689 25134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:05.770359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49718 (local address 127.5.83.62:45997)
0808 04:07:05.770564 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0808 04:07:05.770569 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:07:05.771831 (+ 1262us) server_negotiation.cc:407] Connection header received
0808 04:07:05.773631 (+ 1800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:07:05.773637 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:07:05.773716 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:07:05.773811 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:07:05.775221 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:05.775971 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:05.777454 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:05.777871 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:05.777983 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:05.781235 (+ 3252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:07:05.781239 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:07:05.781242 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:07:05.781275 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:07:05.783862 (+ 2587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:05.784556 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:05.784562 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:05.784566 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:05.784688 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:05.785104 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:05.785108 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:05.785137 (+ 29us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:05.785426 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:07:05.785598 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0808 04:07:05.786305 (+ 707us) server_negotiation.cc:299] Negotiation successful
0808 04:07:05.786473 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 04:07:05.787997 25246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:05.770865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:45997 (local address 127.0.0.1:49718)
0808 04:07:05.771658 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:05.771699 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 04:07:05.773357 (+ 1658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:07:05.774038 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:07:05.774046 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:07:05.774533 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:07:05.775056 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:07:05.775072 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:07:05.776134 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:07:05.776138 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:07:05.777313 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:07:05.777326 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:07:05.778087 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:07:05.778095 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:07:05.778310 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:05.778930 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:07:05.778952 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:07:05.781026 (+ 2074us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:07:05.783987 (+ 2961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:07:05.783995 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:07:05.784009 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:07:05.784442 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:07:05.784794 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:07:05.784799 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:07:05.784802 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:07:05.784988 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:07:05.785573 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:07:05.785580 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:07:05.785916 (+ 336us) client_negotiation.cc:769] Sending connection context
0808 04:07:05.786210 (+ 294us) client_negotiation.cc:240] Negotiation successful
0808 04:07:05.786674 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":392,"threads_started":1}
I20260808 04:07:05.789618 25238 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.83.61" port: 35491 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 04:07:05 dist-test-slave-1l3l krb5kdc[25009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162025, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.61@KRBTEST.COM for kudu/127.5.83.61@KRBTEST.COM
I20260808 04:07:05.803956 25246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:05.790290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.61:35491 (local address 127.0.0.1:51654)
0808 04:07:05.790483 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:05.790503 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 04:07:05.790601 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:07:05.791538 (+ 937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:07:05.791546 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:07:05.791985 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:07:05.792142 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:07:05.792152 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:07:05.793443 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:07:05.793448 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:07:05.794575 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:07:05.794590 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:07:05.795638 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:07:05.795642 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:07:05.795838 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:05.796379 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:07:05.796394 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:07:05.798247 (+ 1853us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:07:05.801845 (+ 3598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:07:05.801849 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:07:05.801851 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:07:05.802169 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:07:05.802535 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:07:05.802538 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:07:05.802540 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:07:05.802668 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:07:05.803275 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:07:05.803279 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:07:05.803393 (+ 114us) client_negotiation.cc:769] Sending connection context
0808 04:07:05.803501 (+ 108us) client_negotiation.cc:240] Negotiation successful
0808 04:07:05.803679 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260808 04:07:05.804319 25247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:05.790548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51654 (local address 127.5.83.61:35491)
0808 04:07:05.791150 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0808 04:07:05.791156 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:07:05.791179 (+ 23us) server_negotiation.cc:407] Connection header received
0808 04:07:05.791226 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:07:05.791231 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:07:05.791378 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:07:05.791494 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:07:05.792284 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:05.793289 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:05.794732 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:05.795512 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:05.795635 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:05.798459 (+ 2824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:07:05.798471 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:07:05.798486 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:07:05.798518 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:07:05.801706 (+ 3188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:05.802298 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:05.802305 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:05.802308 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:05.802441 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:05.802768 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:05.802773 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:05.802775 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:05.803167 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:07:05.803376 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0808 04:07:05.803683 (+ 307us) server_negotiation.cc:299] Negotiation successful
0808 04:07:05.803939 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":466,"thread_start_us":343,"threads_started":1}
I20260808 04:07:05.806555 25238 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.83.60" port: 45383 } has no permanent_uuid. Determining permanent_uuid...
I20260808 04:07:05.807619 25246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:05.807171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.60:45383 (local address 127.0.0.1:46672)
0808 04:07:05.807378 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:05.807498 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.60:45383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260808 04:07:05.807899 25161 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.83.60:45383: connect: Connection refused (error 111)
I20260808 04:07:05.809326 25246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:05.809024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.60:45383 (local address 127.0.0.1:46688)
0808 04:07:05.809153 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:05.809226 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.60:45383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260808 04:07:05.809513 25238 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.83.60:45383: Network error: Client connection negotiation failed: client connection to 127.5.83.60:45383: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.60.keytab.
Entry for principal kudu/127.5.83.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.60.keytab.
I20260808 04:07:05.816358 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161826605655-5452-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161826605655-5452-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161826605655-5452-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161826605655-5452-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.60:45383
--webserver_interface=127.5.83.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.60.keytab
--principal=kudu/127.5.83.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:37863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.62:45997,127.5.83.61:35491,127.5.83.60:45383
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:07:05.858783 25238 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.83.60" port: 45383 } attempt: 1
I20260808 04:07:05.860045 25246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:05.859568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.60:45383 (local address 127.0.0.1:46700)
0808 04:07:05.859778 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:05.859889 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.60:45383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260808 04:07:05.861033 25246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:05.860713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.60:45383 (local address 127.0.0.1:46708)
0808 04:07:05.860862 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:05.860930 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.60:45383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260808 04:07:05.861231 25238 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.83.60:45383: Network error: Client connection negotiation failed: client connection to 127.5.83.60:45383: connect: Connection refused (error 111)
I20260808 04:07:05.929445 25238 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.83.60" port: 45383 } attempt: 2
I20260808 04:07:05.930466 25246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:05.930034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.60:45383 (local address 127.0.0.1:46714)
0808 04:07:05.930214 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:05.930308 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.60:45383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260808 04:07:05.931351 25246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:05.931028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.60:45383 (local address 127.0.0.1:46726)
0808 04:07:05.931162 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:05.931233 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.60:45383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260808 04:07:05.931540 25238 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.83.60:45383: Network error: Client connection negotiation failed: client connection to 127.5.83.60:45383: connect: Connection refused (error 111)
I20260808 04:07:05.991168 25126 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.83.61" port: 35491 } attempt: 5
Aug 08 04:07:05 dist-test-slave-1l3l krb5kdc[25009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162025, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.62@KRBTEST.COM for kudu/127.5.83.61@KRBTEST.COM
I20260808 04:07:06.004209 25133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:05.991991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.61:35491 (local address 127.0.0.1:51668)
0808 04:07:05.992201 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:05.992221 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 04:07:05.992411 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:07:05.992774 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:07:05.992779 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:07:05.993216 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:07:05.993421 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:07:05.993431 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:07:05.994416 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:07:05.994421 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:07:05.995259 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:07:05.995271 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:07:05.996073 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:07:05.996079 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:07:05.996276 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:05.996940 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:07:05.996958 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:07:05.998959 (+ 2001us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:07:06.001806 (+ 2847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:07:06.001812 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:07:06.001817 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:07:06.002200 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:07:06.002547 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:07:06.002552 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:07:06.002556 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:07:06.002693 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:07:06.003201 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:07:06.003207 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:07:06.003377 (+ 170us) client_negotiation.cc:769] Sending connection context
0808 04:07:06.003505 (+ 128us) client_negotiation.cc:240] Negotiation successful
0808 04:07:06.003810 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260808 04:07:06.004209 25247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:05.992035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51668 (local address 127.5.83.61:35491)
0808 04:07:05.992206 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0808 04:07:05.992213 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:07:05.992308 (+ 95us) server_negotiation.cc:407] Connection header received
0808 04:07:05.992548 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:07:05.992552 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:07:05.992633 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:07:05.992720 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:07:05.993580 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:05.994273 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:05.995417 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:05.995909 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:05.996030 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:05.999135 (+ 3105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:07:05.999140 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:07:05.999143 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:07:05.999174 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:07:06.001674 (+ 2500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:06.002310 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:06.002315 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:06.002317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:06.002442 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:06.002793 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:06.002798 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:06.002801 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:06.003085 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:07:06.003220 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0808 04:07:06.003625 (+ 405us) server_negotiation.cc:299] Negotiation successful
0808 04:07:06.003889 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260808 04:07:06.005340 25126 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.83.60" port: 45383 } has no permanent_uuid. Determining permanent_uuid...
I20260808 04:07:06.006495 25133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:06.006094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.60:45383 (local address 127.0.0.1:46736)
0808 04:07:06.006244 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:06.006360 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.60:45383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260808 04:07:06.007388 25133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:06.007070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.60:45383 (local address 127.0.0.1:46750)
0808 04:07:06.007198 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:06.007284 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.60:45383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260808 04:07:06.007570 25126 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.83.60:45383: Network error: Client connection negotiation failed: client connection to 127.5.83.60:45383: connect: Connection refused (error 111)
I20260808 04:07:06.022742 25238 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.83.60" port: 45383 } attempt: 3
I20260808 04:07:06.023945 25246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:06.023329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.60:45383 (local address 127.0.0.1:46752)
0808 04:07:06.023561 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:06.023679 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.60:45383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260808 04:07:06.025141 25246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:06.024642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.60:45383 (local address 127.0.0.1:46758)
0808 04:07:06.024883 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:06.024995 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.60:45383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
W20260808 04:07:06.025362 25238 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.83.60:45383: Network error: Client connection negotiation failed: client connection to 127.5.83.60:45383: connect: Connection refused (error 111)
I20260808 04:07:06.058790 25126 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.83.60" port: 45383 } attempt: 1
I20260808 04:07:06.059919 25133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:06.059425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.60:45383 (local address 127.0.0.1:46760)
0808 04:07:06.059605 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:06.059695 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.60:45383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260808 04:07:06.060904 25133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:06.060597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.60:45383 (local address 127.0.0.1:46762)
0808 04:07:06.060728 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:06.060800 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.60:45383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260808 04:07:06.061105 25126 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.83.60:45383: Network error: Client connection negotiation failed: client connection to 127.5.83.60:45383: connect: Connection refused (error 111)
I20260808 04:07:06.129354 25126 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.83.60" port: 45383 } attempt: 2
I20260808 04:07:06.130555 25133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:06.130049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.60:45383 (local address 127.0.0.1:46768)
0808 04:07:06.130274 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:06.130365 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.60:45383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260808 04:07:06.131597 25133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:06.131200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.60:45383 (local address 127.0.0.1:46770)
0808 04:07:06.131333 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:06.131459 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.60:45383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260808 04:07:06.131924 25126 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.83.60:45383: Network error: Client connection negotiation failed: client connection to 127.5.83.60:45383: connect: Connection refused (error 111)
W20260808 04:07:06.138962 25252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:07:06.139200 25252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:07:06.139293 25252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:07:06.146237 25252 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:07:06.146328 25252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:07:06.146382 25252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:07:06.146448 25252 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:07:06.146513 25252 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:07:06.155941 25252 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:37863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161826605655-5452-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161826605655-5452-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.62:45997,127.5.83.61:35491,127.5.83.60:45383
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.60.keytab
--principal=kudu/127.5.83.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.60:45383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161826605655-5452-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.5.83.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161826605655-5452-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:07:06.157162 25252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:07:06.158339 25252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 04:07:06.168578 25238 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.83.60" port: 45383 } attempt: 4
I20260808 04:07:06.169718 25246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:06.169176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.60:45383 (local address 127.0.0.1:46780)
0808 04:07:06.169421 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:06.169525 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.60:45383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260808 04:07:06.169767 25260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:07:06.169700 25262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:07:06.170215 25266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:07:06.170908 25252 server_base.cc:1034] running on GCE node
I20260808 04:07:06.171013 25246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:06.170637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.60:45383 (local address 127.0.0.1:46786)
0808 04:07:06.170780 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:06.170873 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.60:45383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260808 04:07:06.171231 25238 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.83.60:45383: Network error: Client connection negotiation failed: client connection to 127.5.83.60:45383: connect: Connection refused (error 111)
I20260808 04:07:06.171736 25252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:07:06.173025 25252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:07:06.174234 25252 hybrid_clock.cc:648] HybridClock initialized: now 1786162026174186 us; error 36 us; skew 500 ppm
I20260808 04:07:06.174553 25252 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:07:06 dist-test-slave-1l3l krb5kdc[25009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162026, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:07:06.177610 25252 init.cc:377] Logged in from keytab as kudu/127.5.83.60@KRBTEST.COM (short username kudu)
I20260808 04:07:06.178992 25252 webserver.cc:460] Webserver started at http://127.5.83.60:35529/ using document root <none> and password file <none>
I20260808 04:07:06.179639 25252 fs_manager.cc:359] Metadata directory not provided
I20260808 04:07:06.179780 25252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:07:06.180076 25252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:07:06.182883 25252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161826605655-5452-0/minicluster-data/master-2/data/instance:
uuid: "db958289e5a549bebfe141823c3dac71"
format_stamp: "Formatted at 2026-08-08 04:07:06 on dist-test-slave-1l3l"
server_key: "c6aaa0c7169490b7d94d75d9be0eabc0"
server_key_iv: "487b6e0e0e56ea9fd0c4aad60bba62e3"
server_key_version: "encryptionkey@0"
I20260808 04:07:06.183640 25252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161826605655-5452-0/minicluster-data/master-2/wal/instance:
uuid: "db958289e5a549bebfe141823c3dac71"
format_stamp: "Formatted at 2026-08-08 04:07:06 on dist-test-slave-1l3l"
server_key: "c6aaa0c7169490b7d94d75d9be0eabc0"
server_key_iv: "487b6e0e0e56ea9fd0c4aad60bba62e3"
server_key_version: "encryptionkey@0"
I20260808 04:07:06.188377 25252 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260808 04:07:06.191548 25275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:07:06.192790 25252 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 04:07:06.192958 25252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161826605655-5452-0/minicluster-data/master-2/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161826605655-5452-0/minicluster-data/master-2/wal
uuid: "db958289e5a549bebfe141823c3dac71"
format_stamp: "Formatted at 2026-08-08 04:07:06 on dist-test-slave-1l3l"
server_key: "c6aaa0c7169490b7d94d75d9be0eabc0"
server_key_iv: "487b6e0e0e56ea9fd0c4aad60bba62e3"
server_key_version: "encryptionkey@0"
I20260808 04:07:06.193131 25252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161826605655-5452-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161826605655-5452-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161826605655-5452-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:07:06.206454 25252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:07:06.210107 25252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:07:06.210352 25252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:07:06.227762 25252 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.60:45383
I20260808 04:07:06.227769 25346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.60:45383 every 8 connection(s)
I20260808 04:07:06.229161 25252 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161826605655-5452-0/minicluster-data/master-2/data/info.pb
I20260808 04:07:06.232867 25349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:07:06.238013 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 25252
I20260808 04:07:06.238194 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161826605655-5452-0/minicluster-data/master-2/wal/instance
I20260808 04:07:06.238178 25126 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.83.60" port: 45383 } attempt: 3
I20260808 04:07:06.238713 5452 external_mini_cluster.cc:1428] Setting key ec808aed3cbeba9df3675ff3942481ea
I20260808 04:07:06.239603 25349 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.83.62" port: 45997 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 04:07:06 dist-test-slave-1l3l krb5kdc[25009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162025, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.62@KRBTEST.COM for kudu/127.5.83.60@KRBTEST.COM
Aug 08 04:07:06 dist-test-slave-1l3l krb5kdc[25009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162026, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.60@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
Aug 08 04:07:06 dist-test-slave-1l3l krb5kdc[25009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.60@KRBTEST.COM
I20260808 04:07:06.262969 25133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:06.238854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.60:45383 (local address 127.0.0.1:46790)
0808 04:07:06.239222 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:06.239245 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 04:07:06.239379 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:07:06.246225 (+ 6846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:07:06.246230 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:07:06.246764 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:07:06.246997 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:07:06.247012 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:07:06.249769 (+ 2757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:07:06.249775 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:07:06.250677 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:07:06.250688 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:07:06.251631 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:07:06.251637 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:07:06.251844 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:06.252429 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:07:06.252446 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:07:06.254934 (+ 2488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:07:06.259805 (+ 4871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:07:06.259811 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:07:06.259815 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:07:06.260355 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:07:06.260870 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:07:06.260876 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:07:06.260880 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:07:06.261045 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:07:06.261998 (+ 953us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:07:06.262002 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:07:06.262186 (+ 184us) client_negotiation.cc:769] Sending connection context
0808 04:07:06.262342 (+ 156us) client_negotiation.cc:240] Negotiation successful
0808 04:07:06.262552 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178}
I20260808 04:07:06.263607 25134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:06.242999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49734 (local address 127.5.83.62:45997)
0808 04:07:06.243175 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0808 04:07:06.243182 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:07:06.246928 (+ 3746us) server_negotiation.cc:407] Connection header received
0808 04:07:06.247215 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:07:06.247220 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:07:06.247293 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:07:06.247359 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:07:06.248910 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:06.249653 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:06.251679 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:06.252414 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:06.252529 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:06.257316 (+ 4787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:07:06.257321 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:07:06.257324 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:07:06.257354 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:07:06.260143 (+ 2789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:06.261063 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:06.261069 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:06.261073 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:06.261262 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:06.261679 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:06.261683 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:06.261687 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:06.262061 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:07:06.262203 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0808 04:07:06.262829 (+ 626us) server_negotiation.cc:299] Negotiation successful
0808 04:07:06.263277 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 04:07:06.264545 25355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:06.243442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:45997 (local address 127.0.0.1:49734)
0808 04:07:06.246796 (+ 3354us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:06.246834 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0808 04:07:06.247054 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:07:06.247602 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:07:06.247608 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:07:06.248394 (+ 786us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:07:06.248751 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:07:06.248764 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:07:06.249838 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:07:06.249843 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:07:06.251511 (+ 1668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:07:06.251526 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:07:06.253612 (+ 2086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:07:06.253620 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:07:06.253780 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:06.254347 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:07:06.254374 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:07:06.257129 (+ 2755us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:07:06.260308 (+ 3179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:07:06.260316 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:07:06.260325 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:07:06.260967 (+ 642us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:07:06.261373 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:07:06.261378 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:07:06.261381 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:07:06.261555 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:07:06.262205 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:07:06.262213 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:07:06.262422 (+ 209us) client_negotiation.cc:769] Sending connection context
0808 04:07:06.262712 (+ 290us) client_negotiation.cc:240] Negotiation successful
0808 04:07:06.263210 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3136,"mutex_wait_us":49,"thread_start_us":3010,"threads_started":1}
I20260808 04:07:06.264591 25353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:06.239908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46790 (local address 127.5.83.60:45383)
0808 04:07:06.242651 (+ 2743us) server_negotiation.cc:206] Beginning negotiation
0808 04:07:06.242667 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 04:07:06.242716 (+ 49us) server_negotiation.cc:407] Connection header received
0808 04:07:06.245321 (+ 2605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:07:06.245351 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:07:06.245832 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:07:06.247244 (+ 1412us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:07:06.247634 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:06.249626 (+ 1992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:06.250807 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:06.251490 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:06.251653 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:06.255138 (+ 3485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:07:06.255146 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:07:06.255156 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:07:06.255184 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:07:06.259678 (+ 4494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:06.260517 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:06.260525 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:06.260529 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:06.260734 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:06.261210 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:06.261215 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:06.261218 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:06.261841 (+ 623us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:07:06.262022 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0808 04:07:06.263429 (+ 1407us) server_negotiation.cc:299] Negotiation successful
0808 04:07:06.263654 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":2527,"thread_start_us":2272,"threads_started":1}
I20260808 04:07:06.264964 25354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:06.239941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46792 (local address 127.5.83.60:45383)
0808 04:07:06.247438 (+ 7497us) server_negotiation.cc:206] Beginning negotiation
0808 04:07:06.247443 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:07:06.247466 (+ 23us) server_negotiation.cc:407] Connection header received
0808 04:07:06.247507 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:07:06.247511 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:07:06.247570 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:07:06.247666 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:07:06.249502 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:06.250742 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:06.252619 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:06.253218 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:06.253338 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:06.257854 (+ 4516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:07:06.257861 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:07:06.257866 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:07:06.257914 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:07:06.261302 (+ 3388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:06.262070 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:06.262075 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:06.262078 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:06.262233 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:06.262672 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:06.262677 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:06.262681 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:06.263014 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:07:06.264269 (+ 1255us) server_negotiation.cc:1035] Waiting for connection context
0808 04:07:06.264384 (+ 115us) server_negotiation.cc:299] Negotiation successful
0808 04:07:06.264604 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":7378,"spinlock_wait_cycles":49536,"thread_start_us":7090,"threads_started":1}
I20260808 04:07:06.266691 25349 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.83.61" port: 35491 } has no permanent_uuid. Determining permanent_uuid...
I20260808 04:07:06.272447 25126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e7ccafc1d50430980f490ecdc271ca0: Bootstrap starting.
Aug 08 04:07:06 dist-test-slave-1l3l krb5kdc[25009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162026, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.60@KRBTEST.COM for kudu/127.5.83.61@KRBTEST.COM
Aug 08 04:07:06 dist-test-slave-1l3l krb5kdc[25009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:07:06.276296 25126 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e7ccafc1d50430980f490ecdc271ca0: Neither blocks nor log segments found. Creating new log.
I20260808 04:07:06.277617 25126 log.cc:824] T 00000000000000000000000000000000 P 7e7ccafc1d50430980f490ecdc271ca0: Log is configured to *not* fsync() on all Append() calls
I20260808 04:07:06.280471 25126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e7ccafc1d50430980f490ecdc271ca0: No bootstrap required, opened a new log
I20260808 04:07:06.283020 25355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:06.267369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.61:35491 (local address 127.0.0.1:51670)
0808 04:07:06.267533 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:06.267552 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 04:07:06.267640 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:07:06.268572 (+ 932us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:07:06.268576 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:07:06.268985 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:07:06.269146 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:07:06.269156 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:07:06.270217 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:07:06.270222 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:07:06.271302 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:07:06.271316 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:07:06.272839 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:07:06.272845 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:07:06.273049 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:06.273701 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:07:06.273721 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:07:06.276759 (+ 3038us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:07:06.280566 (+ 3807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:07:06.280571 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:07:06.280574 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:07:06.280925 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:07:06.281355 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:07:06.281360 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:07:06.281364 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:07:06.281543 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:07:06.282162 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:07:06.282168 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:07:06.282312 (+ 144us) client_negotiation.cc:769] Sending connection context
0808 04:07:06.282447 (+ 135us) client_negotiation.cc:240] Negotiation successful
0808 04:07:06.282694 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260808 04:07:06.283030 25134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:06.267974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49744 (local address 127.5.83.62:45997)
0808 04:07:06.268194 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0808 04:07:06.268200 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:07:06.268215 (+ 15us) server_negotiation.cc:407] Connection header received
0808 04:07:06.268255 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:07:06.268259 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:07:06.268318 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:07:06.268423 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:07:06.269567 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:06.270698 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:06.272248 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:06.272717 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:06.272830 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:06.276954 (+ 4124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:07:06.276958 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:07:06.276961 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:07:06.276989 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:07:06.280363 (+ 3374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:06.281131 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:06.281136 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:06.281139 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:06.281269 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:06.281635 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:06.281641 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:06.281644 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:06.281928 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:07:06.282209 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0808 04:07:06.282485 (+ 276us) server_negotiation.cc:299] Negotiation successful
0808 04:07:06.282690 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260808 04:07:06.283968 25247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:06.267527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51670 (local address 127.5.83.61:35491)
0808 04:07:06.267728 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0808 04:07:06.267735 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:07:06.267752 (+ 17us) server_negotiation.cc:407] Connection header received
0808 04:07:06.267807 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:07:06.267812 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:07:06.267883 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:07:06.267975 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:07:06.269315 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:06.270073 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:06.271462 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:06.272152 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:06.272288 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:06.276972 (+ 4684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:07:06.276978 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:07:06.276983 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:07:06.277020 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:07:06.280428 (+ 3408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:06.281051 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:06.281056 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:06.281059 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:06.281220 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:06.281683 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:06.281688 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:06.281691 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:06.282022 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:07:06.283246 (+ 1224us) server_negotiation.cc:1035] Waiting for connection context
0808 04:07:06.283390 (+ 144us) server_negotiation.cc:299] Negotiation successful
0808 04:07:06.283597 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260808 04:07:06.284912 25349 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.83.60" port: 45383 } has no permanent_uuid. Determining permanent_uuid...
I20260808 04:07:06.290166 25126 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e7ccafc1d50430980f490ecdc271ca0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7ccafc1d50430980f490ecdc271ca0" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 45997 } } peers { permanent_uuid: "2c936fbf17904ddfac4994949dc4065a" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 35491 } } peers { permanent_uuid: "db958289e5a549bebfe141823c3dac71" member_type: VOTER last_known_addr { host: "127.5.83.60" port: 45383 } }
I20260808 04:07:06.290468 25126 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e7ccafc1d50430980f490ecdc271ca0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:07:06.290587 25126 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e7ccafc1d50430980f490ecdc271ca0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e7ccafc1d50430980f490ecdc271ca0, State: Initialized, Role: FOLLOWER
I20260808 04:07:06.291117 25126 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e7ccafc1d50430980f490ecdc271ca0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7ccafc1d50430980f490ecdc271ca0" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 45997 } } peers { permanent_uuid: "2c936fbf17904ddfac4994949dc4065a" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 35491 } } peers { permanent_uuid: "db958289e5a549bebfe141823c3dac71" member_type: VOTER last_known_addr { host: "127.5.83.60" port: 45383 } }
Aug 08 04:07:06 dist-test-slave-1l3l krb5kdc[25009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162026, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.60@KRBTEST.COM for kudu/127.5.83.60@KRBTEST.COM
Aug 08 04:07:06 dist-test-slave-1l3l krb5kdc[25009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.61@KRBTEST.COM
I20260808 04:07:06.300238 25355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:06.285641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.60:45383 (local address 127.0.0.1:46806)
0808 04:07:06.285807 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:06.285827 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 04:07:06.285938 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:07:06.286425 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:07:06.286430 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:07:06.286851 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:07:06.287043 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:07:06.287055 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:07:06.288000 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:07:06.288005 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:07:06.289239 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:07:06.289253 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:07:06.290133 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:07:06.290139 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:07:06.290312 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:06.290865 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:07:06.291101 (+ 236us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:07:06.293584 (+ 2483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:07:06.297666 (+ 4082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:07:06.297670 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:07:06.297674 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:07:06.298108 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:07:06.298501 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:07:06.298505 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:07:06.298509 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:07:06.298648 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:07:06.299303 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:07:06.299307 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:07:06.299457 (+ 150us) client_negotiation.cc:769] Sending connection context
0808 04:07:06.299606 (+ 149us) client_negotiation.cc:240] Negotiation successful
0808 04:07:06.299905 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260808 04:07:06.301107 25354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:06.285990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46806 (local address 127.5.83.60:45383)
0808 04:07:06.286161 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0808 04:07:06.286166 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:07:06.286183 (+ 17us) server_negotiation.cc:407] Connection header received
0808 04:07:06.286223 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:07:06.286227 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:07:06.286285 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:07:06.286377 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:07:06.287163 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:06.287883 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:06.289389 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:06.289983 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:06.290107 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:06.293757 (+ 3650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:07:06.293763 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:07:06.293768 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:07:06.293801 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:07:06.297533 (+ 3732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:06.298234 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:06.298239 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:06.298242 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:06.298374 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:06.298755 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:06.298758 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:06.298761 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:06.299107 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:07:06.299254 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0808 04:07:06.300627 (+ 1373us) server_negotiation.cc:299] Negotiation successful
0808 04:07:06.300826 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 04:07:06.302313 25247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:06.284705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51672 (local address 127.5.83.61:35491)
0808 04:07:06.285314 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0808 04:07:06.285318 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:07:06.285333 (+ 15us) server_negotiation.cc:407] Connection header received
0808 04:07:06.285368 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:07:06.285372 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:07:06.285434 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:07:06.285524 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:07:06.287050 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:06.288121 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:06.289731 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:06.290165 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:06.290278 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:06.295377 (+ 5099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:07:06.295384 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:07:06.295388 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:07:06.295424 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:07:06.298746 (+ 3322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:06.299523 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:06.299528 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:06.299531 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:06.299746 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:06.300198 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:06.300203 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:06.300206 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:06.300530 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:07:06.300778 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0808 04:07:06.301797 (+ 1019us) server_negotiation.cc:299] Negotiation successful
0808 04:07:06.302022 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 04:07:06.305907 25349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db958289e5a549bebfe141823c3dac71: Bootstrap starting.
I20260808 04:07:06.308526 5452 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:07:06.309537 25349 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db958289e5a549bebfe141823c3dac71: Neither blocks nor log segments found. Creating new log.
I20260808 04:07:06.309703 25360 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e7ccafc1d50430980f490ecdc271ca0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7ccafc1d50430980f490ecdc271ca0" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 45997 } } peers { permanent_uuid: "2c936fbf17904ddfac4994949dc4065a" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 35491 } } peers { permanent_uuid: "db958289e5a549bebfe141823c3dac71" member_type: VOTER last_known_addr { host: "127.5.83.60" port: 45383 } } }
I20260808 04:07:06.310176 25360 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e7ccafc1d50430980f490ecdc271ca0 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:07:06.310755 25126 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e7ccafc1d50430980f490ecdc271ca0 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:07:06.310736 25349 log.cc:824] T 00000000000000000000000000000000 P db958289e5a549bebfe141823c3dac71: Log is configured to *not* fsync() on all Append() calls
I20260808 04:07:06.313985 25349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db958289e5a549bebfe141823c3dac71: No bootstrap required, opened a new log
I20260808 04:07:06.325320 25349 raft_consensus.cc:348] T 00000000000000000000000000000000 P db958289e5a549bebfe141823c3dac71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7ccafc1d50430980f490ecdc271ca0" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 45997 } } peers { permanent_uuid: "2c936fbf17904ddfac4994949dc4065a" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 35491 } } peers { permanent_uuid: "db958289e5a549bebfe141823c3dac71" member_type: VOTER last_known_addr { host: "127.5.83.60" port: 45383 } }
I20260808 04:07:06.325742 25349 raft_consensus.cc:374] T 00000000000000000000000000000000 P db958289e5a549bebfe141823c3dac71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:07:06.325883 25349 raft_consensus.cc:729] T 00000000000000000000000000000000 P db958289e5a549bebfe141823c3dac71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db958289e5a549bebfe141823c3dac71, State: Initialized, Role: FOLLOWER
I20260808 04:07:06.327028 25349 consensus_queue.cc:260] T 00000000000000000000000000000000 P db958289e5a549bebfe141823c3dac71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7ccafc1d50430980f490ecdc271ca0" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 45997 } } peers { permanent_uuid: "2c936fbf17904ddfac4994949dc4065a" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 35491 } } peers { permanent_uuid: "db958289e5a549bebfe141823c3dac71" member_type: VOTER last_known_addr { host: "127.5.83.60" port: 45383 } }
I20260808 04:07:06.330089 25383 sys_catalog.cc:455] T 00000000000000000000000000000000 P db958289e5a549bebfe141823c3dac71 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7ccafc1d50430980f490ecdc271ca0" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 45997 } } peers { permanent_uuid: "2c936fbf17904ddfac4994949dc4065a" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 35491 } } peers { permanent_uuid: "db958289e5a549bebfe141823c3dac71" member_type: VOTER last_known_addr { host: "127.5.83.60" port: 45383 } } }
I20260808 04:07:06.330493 25383 sys_catalog.cc:458] T 00000000000000000000000000000000 P db958289e5a549bebfe141823c3dac71 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:07:06.332175 25349 sys_catalog.cc:564] T 00000000000000000000000000000000 P db958289e5a549bebfe141823c3dac71 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:07:06.336227 25134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:06.317198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49756 (local address 127.5.83.62:45997)
0808 04:07:06.317352 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0808 04:07:06.317358 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:07:06.317925 (+ 567us) server_negotiation.cc:407] Connection header received
0808 04:07:06.318177 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:07:06.318182 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:07:06.318257 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:07:06.318320 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:07:06.320636 (+ 2316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:06.321788 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:06.324040 (+ 2252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:06.324677 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:06.324776 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:06.328741 (+ 3965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:07:06.328747 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:07:06.328752 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:07:06.328791 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:07:06.332310 (+ 3519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:06.333150 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:06.333156 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:06.333159 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:06.333350 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:06.333816 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:06.333821 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:06.333825 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:06.334185 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:07:06.334451 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0808 04:07:06.335512 (+ 1061us) server_negotiation.cc:299] Negotiation successful
0808 04:07:06.335771 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 04:07:06.336863 25247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:06.318140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51686 (local address 127.5.83.61:35491)
0808 04:07:06.318303 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0808 04:07:06.318309 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:07:06.319365 (+ 1056us) server_negotiation.cc:407] Connection header received
0808 04:07:06.319630 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:07:06.319636 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:07:06.319739 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:07:06.319808 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:07:06.321991 (+ 2183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:06.323115 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:06.325449 (+ 2334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:06.326081 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:06.326177 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:06.329792 (+ 3615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:07:06.329798 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:07:06.329803 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:07:06.329843 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:07:06.333513 (+ 3670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:06.334395 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:06.334401 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:06.334405 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:06.334574 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:06.335083 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:06.335089 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:06.335092 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:06.335482 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:07:06.335696 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0808 04:07:06.336286 (+ 590us) server_negotiation.cc:299] Negotiation successful
0808 04:07:06.336502 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 04:07:06.355086 25354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:06.324028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46812 (local address 127.5.83.60:45383)
0808 04:07:06.324180 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0808 04:07:06.324185 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:07:06.335320 (+ 11135us) server_negotiation.cc:407] Connection header received
0808 04:07:06.338733 (+ 3413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:07:06.338739 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:07:06.338821 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:07:06.338893 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:07:06.341142 (+ 2249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:06.342300 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:06.344186 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:06.344922 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:06.345039 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:06.347654 (+ 2615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:07:06.347660 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:07:06.347665 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:07:06.347723 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:07:06.351199 (+ 3476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:06.352152 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:06.352158 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:06.352162 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:06.352308 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:06.352764 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:06.352769 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:06.352773 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:06.353159 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:07:06.354326 (+ 1167us) server_negotiation.cc:1035] Waiting for connection context
0808 04:07:06.354469 (+ 143us) server_negotiation.cc:299] Negotiation successful
0808 04:07:06.354709 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 04:07:06.349419 25027 master_runner.cc:186] Error getting master registration for 127.5.83.61:35491: OK, instance_id { permanent_uuid: "2c936fbf17904ddfac4994949dc4065a" instance_seqno: 1786162025749190 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 04:07:06.363080 25027 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260808 04:07:06.366483 25399 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7e7ccafc1d50430980f490ecdc271ca0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 04:07:06.366603 25399 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260808 04:07:06.372730 25407 catalog_manager.cc:1558] T 00000000000000000000000000000000 P db958289e5a549bebfe141823c3dac71: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 04:07:06.372915 25407 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 04:07:06.374269 25252 master_runner.cc:186] Error getting master registration for 127.5.83.61:35491: OK, instance_id { permanent_uuid: "2c936fbf17904ddfac4994949dc4065a" instance_seqno: 1786162025749190 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 04:07:06.386032 25252 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 04:07:06.415243 25360 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e7ccafc1d50430980f490ecdc271ca0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:07:06.415542 25360 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e7ccafc1d50430980f490ecdc271ca0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7ccafc1d50430980f490ecdc271ca0" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 45997 } } peers { permanent_uuid: "2c936fbf17904ddfac4994949dc4065a" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 35491 } } peers { permanent_uuid: "db958289e5a549bebfe141823c3dac71" member_type: VOTER last_known_addr { host: "127.5.83.60" port: 45383 } }
I20260808 04:07:06.417176 25360 leader_election.cc:288] T 00000000000000000000000000000000 P 7e7ccafc1d50430980f490ecdc271ca0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2c936fbf17904ddfac4994949dc4065a (127.5.83.61:35491), db958289e5a549bebfe141823c3dac71 (127.5.83.60:45383)
I20260808 04:07:06.418121 25315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7e7ccafc1d50430980f490ecdc271ca0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db958289e5a549bebfe141823c3dac71" is_pre_election: true
I20260808 04:07:06.418752 25315 raft_consensus.cc:2454] T 00000000000000000000000000000000 P db958289e5a549bebfe141823c3dac71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e7ccafc1d50430980f490ecdc271ca0 in term 0.
I20260808 04:07:06.419377 25051 leader_election.cc:302] T 00000000000000000000000000000000 P 7e7ccafc1d50430980f490ecdc271ca0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e7ccafc1d50430980f490ecdc271ca0, db958289e5a549bebfe141823c3dac71; no voters:
I20260808 04:07:06.419838 25360 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e7ccafc1d50430980f490ecdc271ca0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:07:06.419996 25360 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e7ccafc1d50430980f490ecdc271ca0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:07:06.420089 25360 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e7ccafc1d50430980f490ecdc271ca0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:07:06.424142 25360 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e7ccafc1d50430980f490ecdc271ca0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7ccafc1d50430980f490ecdc271ca0" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 45997 } } peers { permanent_uuid: "2c936fbf17904ddfac4994949dc4065a" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 35491 } } peers { permanent_uuid: "db958289e5a549bebfe141823c3dac71" member_type: VOTER last_known_addr { host: "127.5.83.60" port: 45383 } }
I20260808 04:07:06.426592 25315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7e7ccafc1d50430980f490ecdc271ca0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db958289e5a549bebfe141823c3dac71"
I20260808 04:07:06.426792 25315 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db958289e5a549bebfe141823c3dac71 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:07:06.430600 25315 raft_consensus.cc:2454] T 00000000000000000000000000000000 P db958289e5a549bebfe141823c3dac71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e7ccafc1d50430980f490ecdc271ca0 in term 1.
I20260808 04:07:06.430998 25051 leader_election.cc:302] T 00000000000000000000000000000000 P 7e7ccafc1d50430980f490ecdc271ca0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e7ccafc1d50430980f490ecdc271ca0, db958289e5a549bebfe141823c3dac71; no voters:
I20260808 04:07:06.431777 25360 leader_election.cc:288] T 00000000000000000000000000000000 P 7e7ccafc1d50430980f490ecdc271ca0 [CANDIDATE]: Term 1 election: Requested vote from peers 2c936fbf17904ddfac4994949dc4065a (127.5.83.61:35491), db958289e5a549bebfe141823c3dac71 (127.5.83.60:45383)
I20260808 04:07:06.432165 25410 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e7ccafc1d50430980f490ecdc271ca0 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:07:06.432420 25410 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e7ccafc1d50430980f490ecdc271ca0 [term 1 LEADER]: Becoming Leader. State: Replica: 7e7ccafc1d50430980f490ecdc271ca0, State: Running, Role: LEADER
I20260808 04:07:06.433077 25410 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e7ccafc1d50430980f490ecdc271ca0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7ccafc1d50430980f490ecdc271ca0" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 45997 } } peers { permanent_uuid: "2c936fbf17904ddfac4994949dc4065a" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 35491 } } peers { permanent_uuid: "db958289e5a549bebfe141823c3dac71" member_type: VOTER last_known_addr { host: "127.5.83.60" port: 45383 } }
I20260808 04:07:06.445228 25360 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e7ccafc1d50430980f490ecdc271ca0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e7ccafc1d50430980f490ecdc271ca0. Latest consensus state: current_term: 1 leader_uuid: "7e7ccafc1d50430980f490ecdc271ca0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7ccafc1d50430980f490ecdc271ca0" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 45997 } } peers { permanent_uuid: "2c936fbf17904ddfac4994949dc4065a" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 35491 } } peers { permanent_uuid: "db958289e5a549bebfe141823c3dac71" member_type: VOTER last_known_addr { host: "127.5.83.60" port: 45383 } } }
I20260808 04:07:06.445818 25360 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e7ccafc1d50430980f490ecdc271ca0 [sys.catalog]: This master's current role is: LEADER
I20260808 04:07:06.446977 25413 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:07:06.449200 25413 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260808 04:07:06.462239 25287 tablet.cc:2369] T 00000000000000000000000000000000 P db958289e5a549bebfe141823c3dac71: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:07:06.464673 25315 raft_consensus.cc:1261] T 00000000000000000000000000000000 P db958289e5a549bebfe141823c3dac71 [term 1 FOLLOWER]: Refusing update from remote peer 7e7ccafc1d50430980f490ecdc271ca0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:07:06.465888 25410 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7e7ccafc1d50430980f490ecdc271ca0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db958289e5a549bebfe141823c3dac71" member_type: VOTER last_known_addr { host: "127.5.83.60" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:07:06.470490 25238 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.83.60" port: 45383 } attempt: 5
Aug 08 04:07:06 dist-test-slave-1l3l krb5kdc[25009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162025, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.61@KRBTEST.COM for kudu/127.5.83.60@KRBTEST.COM
I20260808 04:07:06.491597 25246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:06.471032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.60:45383 (local address 127.0.0.1:46824)
0808 04:07:06.471383 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:06.471404 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 04:07:06.471509 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:07:06.472087 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:07:06.472092 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:07:06.472597 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:07:06.472797 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:07:06.472810 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:07:06.474385 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:07:06.474390 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:07:06.475868 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:07:06.475891 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:07:06.477269 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:07:06.477275 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:07:06.477471 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:06.478228 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:07:06.478251 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:07:06.481059 (+ 2808us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:07:06.487035 (+ 5976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:07:06.487040 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:07:06.487045 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:07:06.487562 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:07:06.488265 (+ 703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:07:06.488270 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:07:06.488298 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:07:06.488578 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:07:06.489805 (+ 1227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:07:06.489810 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:07:06.490184 (+ 374us) client_negotiation.cc:769] Sending connection context
0808 04:07:06.490424 (+ 240us) client_negotiation.cc:240] Negotiation successful
0808 04:07:06.490927 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219}
I20260808 04:07:06.495222 25354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:06.471463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46824 (local address 127.5.83.60:45383)
0808 04:07:06.471630 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0808 04:07:06.471635 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:07:06.471653 (+ 18us) server_negotiation.cc:407] Connection header received
0808 04:07:06.471719 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:07:06.471724 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:07:06.471798 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:07:06.471910 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:07:06.473003 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:06.474188 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:06.476071 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:06.476810 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:06.476936 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:06.481247 (+ 4311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:07:06.481254 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:07:06.481259 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:07:06.481296 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:07:06.486837 (+ 5541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:06.487751 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:06.487758 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:06.487762 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:06.488044 (+ 282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:06.488803 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:06.488840 (+ 37us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:06.488844 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:06.489601 (+ 757us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:07:06.492241 (+ 2640us) server_negotiation.cc:1035] Waiting for connection context
0808 04:07:06.492720 (+ 479us) server_negotiation.cc:299] Negotiation successful
0808 04:07:06.493559 (+ 839us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 04:07:06.501387 25238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c936fbf17904ddfac4994949dc4065a: Bootstrap starting.
I20260808 04:07:06.505254 25238 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c936fbf17904ddfac4994949dc4065a: Neither blocks nor log segments found. Creating new log.
I20260808 04:07:06.506546 25238 log.cc:824] T 00000000000000000000000000000000 P 2c936fbf17904ddfac4994949dc4065a: Log is configured to *not* fsync() on all Append() calls
I20260808 04:07:06.516453 25238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c936fbf17904ddfac4994949dc4065a: No bootstrap required, opened a new log
I20260808 04:07:06.525493 25383 sys_catalog.cc:455] T 00000000000000000000000000000000 P db958289e5a549bebfe141823c3dac71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e7ccafc1d50430980f490ecdc271ca0. Latest consensus state: current_term: 1 leader_uuid: "7e7ccafc1d50430980f490ecdc271ca0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7ccafc1d50430980f490ecdc271ca0" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 45997 } } peers { permanent_uuid: "2c936fbf17904ddfac4994949dc4065a" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 35491 } } peers { permanent_uuid: "db958289e5a549bebfe141823c3dac71" member_type: VOTER last_known_addr { host: "127.5.83.60" port: 45383 } } }
I20260808 04:07:06.525772 25383 sys_catalog.cc:458] T 00000000000000000000000000000000 P db958289e5a549bebfe141823c3dac71 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:07:06.527007 25238 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c936fbf17904ddfac4994949dc4065a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7ccafc1d50430980f490ecdc271ca0" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 45997 } } peers { permanent_uuid: "2c936fbf17904ddfac4994949dc4065a" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 35491 } } peers { permanent_uuid: "db958289e5a549bebfe141823c3dac71" member_type: VOTER last_known_addr { host: "127.5.83.60" port: 45383 } }
I20260808 04:07:06.527393 25238 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c936fbf17904ddfac4994949dc4065a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:07:06.527527 25238 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c936fbf17904ddfac4994949dc4065a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c936fbf17904ddfac4994949dc4065a, State: Initialized, Role: FOLLOWER
I20260808 04:07:06.528347 25238 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c936fbf17904ddfac4994949dc4065a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7ccafc1d50430980f490ecdc271ca0" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 45997 } } peers { permanent_uuid: "2c936fbf17904ddfac4994949dc4065a" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 35491 } } peers { permanent_uuid: "db958289e5a549bebfe141823c3dac71" member_type: VOTER last_known_addr { host: "127.5.83.60" port: 45383 } }
W20260808 04:07:06.529057 25059 tablet.cc:2369] T 00000000000000000000000000000000 P 7e7ccafc1d50430980f490ecdc271ca0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:07:06.531113 25416 mvcc.cc:205] Tried to move back new op lower bound from 7316119660386443264 to 7316119660275527680. Current Snapshot: MvccSnapshot[applied={T|T < 7316119660386443264}]
I20260808 04:07:06.536158 25360 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e7ccafc1d50430980f490ecdc271ca0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7e7ccafc1d50430980f490ecdc271ca0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7ccafc1d50430980f490ecdc271ca0" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 45997 } } peers { permanent_uuid: "2c936fbf17904ddfac4994949dc4065a" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 35491 } } peers { permanent_uuid: "db958289e5a549bebfe141823c3dac71" member_type: VOTER last_known_addr { host: "127.5.83.60" port: 45383 } } }
I20260808 04:07:06.536425 25360 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e7ccafc1d50430980f490ecdc271ca0 [sys.catalog]: This master's current role is: LEADER
I20260808 04:07:06.539388 25383 sys_catalog.cc:455] T 00000000000000000000000000000000 P db958289e5a549bebfe141823c3dac71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7e7ccafc1d50430980f490ecdc271ca0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7ccafc1d50430980f490ecdc271ca0" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 45997 } } peers { permanent_uuid: "2c936fbf17904ddfac4994949dc4065a" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 35491 } } peers { permanent_uuid: "db958289e5a549bebfe141823c3dac71" member_type: VOTER last_known_addr { host: "127.5.83.60" port: 45383 } } }
I20260808 04:07:06.539639 25383 sys_catalog.cc:458] T 00000000000000000000000000000000 P db958289e5a549bebfe141823c3dac71 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:07:06.540373 25413 catalog_manager.cc:1347] Generated new cluster ID: 17a4485d616c4d82aa82e3b6d1db5757
I20260808 04:07:06.540484 25413 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:07:06.542661 25422 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c936fbf17904ddfac4994949dc4065a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7ccafc1d50430980f490ecdc271ca0" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 45997 } } peers { permanent_uuid: "2c936fbf17904ddfac4994949dc4065a" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 35491 } } peers { permanent_uuid: "db958289e5a549bebfe141823c3dac71" member_type: VOTER last_known_addr { host: "127.5.83.60" port: 45383 } } }
I20260808 04:07:06.543315 25422 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c936fbf17904ddfac4994949dc4065a [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:07:06.543998 25238 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c936fbf17904ddfac4994949dc4065a [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:07:06.544296 25194 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c936fbf17904ddfac4994949dc4065a [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:07:06.547847 25196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7e7ccafc1d50430980f490ecdc271ca0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c936fbf17904ddfac4994949dc4065a"
I20260808 04:07:06.550227 25198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7e7ccafc1d50430980f490ecdc271ca0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c936fbf17904ddfac4994949dc4065a" is_pre_election: true
I20260808 04:07:06.552680 25194 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2c936fbf17904ddfac4994949dc4065a [term 1 FOLLOWER]: Refusing update from remote peer 7e7ccafc1d50430980f490ecdc271ca0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:07:06.553404 25360 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7e7ccafc1d50430980f490ecdc271ca0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c936fbf17904ddfac4994949dc4065a" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 35491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:07:06.556468 25413 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:07:06.558077 25413 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:07:06.590340 25413 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e7ccafc1d50430980f490ecdc271ca0: Generated new TSK 0
I20260808 04:07:06.591439 25413 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:07:06.598536 25422 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c936fbf17904ddfac4994949dc4065a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e7ccafc1d50430980f490ecdc271ca0. Latest consensus state: current_term: 1 leader_uuid: "7e7ccafc1d50430980f490ecdc271ca0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7ccafc1d50430980f490ecdc271ca0" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 45997 } } peers { permanent_uuid: "2c936fbf17904ddfac4994949dc4065a" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 35491 } } peers { permanent_uuid: "db958289e5a549bebfe141823c3dac71" member_type: VOTER last_known_addr { host: "127.5.83.60" port: 45383 } } }
I20260808 04:07:06.598853 25422 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c936fbf17904ddfac4994949dc4065a [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:07:06.599376 25434 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c936fbf17904ddfac4994949dc4065a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7e7ccafc1d50430980f490ecdc271ca0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7ccafc1d50430980f490ecdc271ca0" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 45997 } } peers { permanent_uuid: "2c936fbf17904ddfac4994949dc4065a" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 35491 } } peers { permanent_uuid: "db958289e5a549bebfe141823c3dac71" member_type: VOTER last_known_addr { host: "127.5.83.60" port: 45383 } } }
I20260808 04:07:06.599814 25434 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c936fbf17904ddfac4994949dc4065a [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:07:06.600890 25360 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e7ccafc1d50430980f490ecdc271ca0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7e7ccafc1d50430980f490ecdc271ca0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7ccafc1d50430980f490ecdc271ca0" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 45997 } } peers { permanent_uuid: "2c936fbf17904ddfac4994949dc4065a" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 35491 } } peers { permanent_uuid: "db958289e5a549bebfe141823c3dac71" member_type: VOTER last_known_addr { host: "127.5.83.60" port: 45383 } } }
I20260808 04:07:06.601161 25360 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e7ccafc1d50430980f490ecdc271ca0 [sys.catalog]: This master's current role is: LEADER
I20260808 04:07:06.625947 25448 catalog_manager.cc:1259] Loaded cluster ID: 17a4485d616c4d82aa82e3b6d1db5757
I20260808 04:07:06.626058 25448 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2c936fbf17904ddfac4994949dc4065a: loading cluster ID for follower catalog manager: success
I20260808 04:07:06.629689 25448 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2c936fbf17904ddfac4994949dc4065a: acquiring CA information for follower catalog manager: success
I20260808 04:07:06.632143 25448 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2c936fbf17904ddfac4994949dc4065a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 04:07:07.374843 25407 catalog_manager.cc:1259] Loaded cluster ID: 17a4485d616c4d82aa82e3b6d1db5757
I20260808 04:07:07.374955 25407 catalog_manager.cc:1552] T 00000000000000000000000000000000 P db958289e5a549bebfe141823c3dac71: loading cluster ID for follower catalog manager: success
I20260808 04:07:07.377486 25407 catalog_manager.cc:1574] T 00000000000000000000000000000000 P db958289e5a549bebfe141823c3dac71: acquiring CA information for follower catalog manager: success
I20260808 04:07:07.379230 25407 catalog_manager.cc:1602] T 00000000000000000000000000000000 P db958289e5a549bebfe141823c3dac71: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 04:07:07.757043 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 25027
I20260808 04:07:07.758406 25120 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:07:07.868187 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 25027
W20260808 04:07:07.892592 25165 connection.cc:535] server connection from 127.0.0.1:51668 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260808 04:07:07.892974 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 25141
I20260808 04:07:07.893961 25229 generic_service.cc:196] Checking for leaks (request via RPC)
W20260808 04:07:07.892596 25279 connection.cc:535] server connection from 127.0.0.1:46790 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260808 04:07:07.999205 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 25141
I20260808 04:07:08.021775 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 25252
I20260808 04:07:08.022920 25341 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:07:08.127422 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 25252
2026-08-08T04:07:08Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4649 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786161826605655-5452-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:07:08 dist-test-slave-1l3l krb5kdc[25471](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:07:08 dist-test-slave-1l3l krb5kdc[25471](info): set up 2 sockets
Aug 08 04:07:08 dist-test-slave-1l3l krb5kdc[25471](info): commencing operation
krb5kdc: starting...
W20260808 04:07:09.312165 5452 mini_kdc.cc:121] Time spent starting KDC: real 1.140s user 0.003s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786161826605655-5452-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786161826605655-5452-0/krb5kdc/test-user.keytab.
Aug 08 04:07:09 dist-test-slave-1l3l krb5kdc[25471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:07:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:07:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.83.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab.
Entry for principal kudu/127.5.83.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab.
I20260808 04:07:09.542474 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786161826605655-5452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786161826605655-5452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786161826605655-5452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:36991
--webserver_interface=127.5.83.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab
--principal=kudu/127.5.83.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:37269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.62:36991
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:07:09.857687 25490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:07:09.857951 25490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:07:09.858040 25490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:07:09.864965 25490 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:07:09.865061 25490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:07:09.865105 25490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:07:09.865161 25490 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:07:09.865206 25490 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:07:09.874473 25490 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:37269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786161826605655-5452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786161826605655-5452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.62:36991
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab
--principal=kudu/127.5.83.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:36991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786161826605655-5452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:07:09.875628 25490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:07:09.876899 25490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:07:09.887943 25502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:07:09.887996 25500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:07:09.888170 25498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:07:09.888825 25490 server_base.cc:1034] running on GCE node
I20260808 04:07:09.889519 25490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:07:09.890798 25490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:07:09.891995 25490 hybrid_clock.cc:648] HybridClock initialized: now 1786162029891951 us; error 38 us; skew 500 ppm
I20260808 04:07:09.892297 25490 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:07:09 dist-test-slave-1l3l krb5kdc[25471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162029, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:07:09.895181 25490 init.cc:377] Logged in from keytab as kudu/127.5.83.62@KRBTEST.COM (short username kudu)
I20260808 04:07:09.896538 25490 webserver.cc:460] Webserver started at http://127.5.83.62:43387/ using document root <none> and password file <none>
I20260808 04:07:09.897152 25490 fs_manager.cc:359] Metadata directory not provided
I20260808 04:07:09.897265 25490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:07:09.897559 25490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:07:09.900382 25490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786161826605655-5452-0/minicluster-data/master-0/data/instance:
uuid: "fb30b1c35f2d45f4a8f7d77db8fcd397"
format_stamp: "Formatted at 2026-08-08 04:07:09 on dist-test-slave-1l3l"
server_key: "bb0d25e3db7cf9482f691e9fb746983c"
server_key_iv: "7ee98389ab4b4b619c90f50804354546"
server_key_version: "encryptionkey@0"
I20260808 04:07:09.901096 25490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786161826605655-5452-0/minicluster-data/master-0/wal/instance:
uuid: "fb30b1c35f2d45f4a8f7d77db8fcd397"
format_stamp: "Formatted at 2026-08-08 04:07:09 on dist-test-slave-1l3l"
server_key: "bb0d25e3db7cf9482f691e9fb746983c"
server_key_iv: "7ee98389ab4b4b619c90f50804354546"
server_key_version: "encryptionkey@0"
I20260808 04:07:09.905829 25490 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260808 04:07:09.909840 25512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:07:09.911031 25490 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260808 04:07:09.911183 25490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786161826605655-5452-0/minicluster-data/master-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786161826605655-5452-0/minicluster-data/master-0/wal
uuid: "fb30b1c35f2d45f4a8f7d77db8fcd397"
format_stamp: "Formatted at 2026-08-08 04:07:09 on dist-test-slave-1l3l"
server_key: "bb0d25e3db7cf9482f691e9fb746983c"
server_key_iv: "7ee98389ab4b4b619c90f50804354546"
server_key_version: "encryptionkey@0"
I20260808 04:07:09.911351 25490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786161826605655-5452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786161826605655-5452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786161826605655-5452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:07:09.938197 25490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:07:09.942243 25490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:07:09.942515 25490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:07:09.959565 25490 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.62:36991
I20260808 04:07:09.959596 25587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.62:36991 every 8 connection(s)
I20260808 04:07:09.960882 25490 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
I20260808 04:07:09.961261 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 25490
I20260808 04:07:09.961463 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786161826605655-5452-0/minicluster-data/master-0/wal/instance
I20260808 04:07:09.962239 5452 external_mini_cluster.cc:1428] Setting key 91270fc9f156d362054334b59d6cb216
I20260808 04:07:09.965049 25588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 08 04:07:09 dist-test-slave-1l3l krb5kdc[25471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:07:09.977613 25588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb30b1c35f2d45f4a8f7d77db8fcd397: Bootstrap starting.
I20260808 04:07:09.981736 25588 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb30b1c35f2d45f4a8f7d77db8fcd397: Neither blocks nor log segments found. Creating new log.
I20260808 04:07:09.982836 25588 log.cc:824] T 00000000000000000000000000000000 P fb30b1c35f2d45f4a8f7d77db8fcd397: Log is configured to *not* fsync() on all Append() calls
I20260808 04:07:09.986016 25592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:09.964057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35734 (local address 127.5.83.62:36991)
0808 04:07:09.964796 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0808 04:07:09.964807 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 04:07:09.964846 (+ 39us) server_negotiation.cc:407] Connection header received
0808 04:07:09.966429 (+ 1583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:07:09.966451 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:07:09.966788 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:07:09.967091 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:07:09.968905 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:09.970269 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:09.972137 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:09.972779 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:09.972918 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:09.977677 (+ 4759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:07:09.977689 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:07:09.977709 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:07:09.977760 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:07:09.981504 (+ 3744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:09.982217 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:09.982225 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:09.982232 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:09.982452 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:09.982902 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:09.982907 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:09.982911 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:09.983585 (+ 674us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:07:09.983845 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0808 04:07:09.984337 (+ 492us) server_negotiation.cc:299] Negotiation successful
0808 04:07:09.984778 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":330,"threads_started":1}
I20260808 04:07:09.986215 25588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb30b1c35f2d45f4a8f7d77db8fcd397: No bootstrap required, opened a new log
I20260808 04:07:09.989693 5452 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:07:09.994025 25588 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb30b1c35f2d45f4a8f7d77db8fcd397 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb30b1c35f2d45f4a8f7d77db8fcd397" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 36991 } }
I20260808 04:07:09.994345 25588 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb30b1c35f2d45f4a8f7d77db8fcd397 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:07:09.994448 25588 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb30b1c35f2d45f4a8f7d77db8fcd397 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb30b1c35f2d45f4a8f7d77db8fcd397, State: Initialized, Role: FOLLOWER
I20260808 04:07:09.994968 25588 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb30b1c35f2d45f4a8f7d77db8fcd397 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb30b1c35f2d45f4a8f7d77db8fcd397" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 36991 } }
I20260808 04:07:09.995177 25588 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb30b1c35f2d45f4a8f7d77db8fcd397 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:07:09.995311 25588 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb30b1c35f2d45f4a8f7d77db8fcd397 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:07:09.995453 25588 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb30b1c35f2d45f4a8f7d77db8fcd397 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:07:09.997536 25588 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb30b1c35f2d45f4a8f7d77db8fcd397 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb30b1c35f2d45f4a8f7d77db8fcd397" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 36991 } }
I20260808 04:07:09.997983 25588 leader_election.cc:302] T 00000000000000000000000000000000 P fb30b1c35f2d45f4a8f7d77db8fcd397 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb30b1c35f2d45f4a8f7d77db8fcd397; no voters:
I20260808 04:07:09.998517 25588 leader_election.cc:288] T 00000000000000000000000000000000 P fb30b1c35f2d45f4a8f7d77db8fcd397 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:07:09.998634 25594 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb30b1c35f2d45f4a8f7d77db8fcd397 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:07:09.999275 25594 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb30b1c35f2d45f4a8f7d77db8fcd397 [term 1 LEADER]: Becoming Leader. State: Replica: fb30b1c35f2d45f4a8f7d77db8fcd397, State: Running, Role: LEADER
I20260808 04:07:09.999794 25594 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb30b1c35f2d45f4a8f7d77db8fcd397 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb30b1c35f2d45f4a8f7d77db8fcd397" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 36991 } }
I20260808 04:07:10.000530 25588 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb30b1c35f2d45f4a8f7d77db8fcd397 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:07:10.002713 25596 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb30b1c35f2d45f4a8f7d77db8fcd397 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fb30b1c35f2d45f4a8f7d77db8fcd397" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb30b1c35f2d45f4a8f7d77db8fcd397" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 36991 } } }
I20260808 04:07:10.002879 25598 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb30b1c35f2d45f4a8f7d77db8fcd397 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb30b1c35f2d45f4a8f7d77db8fcd397. Latest consensus state: current_term: 1 leader_uuid: "fb30b1c35f2d45f4a8f7d77db8fcd397" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb30b1c35f2d45f4a8f7d77db8fcd397" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 36991 } } }
I20260808 04:07:10.003177 25596 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb30b1c35f2d45f4a8f7d77db8fcd397 [sys.catalog]: This master's current role is: LEADER
I20260808 04:07:10.003175 25598 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb30b1c35f2d45f4a8f7d77db8fcd397 [sys.catalog]: This master's current role is: LEADER
I20260808 04:07:10.004815 25613 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:07:10.009246 25613 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:07:10.017092 25613 catalog_manager.cc:1347] Generated new cluster ID: d2a048244d194d44a7a551ac9d38921c
I20260808 04:07:10.017217 25613 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:07:10.027187 25613 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:07:10.028445 25613 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:07:10.035693 25613 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb30b1c35f2d45f4a8f7d77db8fcd397: Generated new TSK 0
I20260808 04:07:10.036437 25613 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:07:10.049217 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 25490
I20260808 04:07:10.050320 25580 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:07:10.137706 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 25490
2026-08-08T04:07:10Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2008 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:07:10 dist-test-slave-1l3l krb5kdc[25636](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:07:10 dist-test-slave-1l3l krb5kdc[25636](info): set up 2 sockets
Aug 08 04:07:10 dist-test-slave-1l3l krb5kdc[25636](info): commencing operation
krb5kdc: starting...
W20260808 04:07:11.320767 5452 mini_kdc.cc:121] Time spent starting KDC: real 1.141s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/krb5kdc/test-user.keytab.
Aug 08 04:07:11 dist-test-slave-1l3l krb5kdc[25636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:07:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:07:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.83.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab.
Entry for principal kudu/127.5.83.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab.
I20260808 04:07:11.572751 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:35721
--webserver_interface=127.5.83.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab
--principal=kudu/127.5.83.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:39647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.62:35721
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:07:11.889772 25656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:07:11.890041 25656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:07:11.890174 25656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:07:11.897197 25656 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:07:11.897308 25656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:07:11.897387 25656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:07:11.897473 25656 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:07:11.897536 25656 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:07:11.906934 25656 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:39647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.62:35721
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab
--principal=kudu/127.5.83.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:35721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:07:11.908149 25656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:07:11.909418 25656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:07:11.920892 25666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:07:11.920955 25662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:07:11.920989 25661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:07:11.922008 25656 server_base.cc:1034] running on GCE node
I20260808 04:07:11.922698 25656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:07:11.924006 25656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:07:11.925203 25656 hybrid_clock.cc:648] HybridClock initialized: now 1786162031925158 us; error 37 us; skew 500 ppm
I20260808 04:07:11.925509 25656 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:07:11 dist-test-slave-1l3l krb5kdc[25636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162031, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:07:11.931145 25656 init.cc:377] Logged in from keytab as kudu/127.5.83.62@KRBTEST.COM (short username kudu)
I20260808 04:07:11.932564 25656 webserver.cc:460] Webserver started at http://127.5.83.62:34815/ using document root <none> and password file <none>
I20260808 04:07:11.933188 25656 fs_manager.cc:359] Metadata directory not provided
I20260808 04:07:11.933306 25656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:07:11.933588 25656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:07:11.936430 25656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/master-0/data/instance:
uuid: "231726dbe3ec4fd49e58e8c2a800ed19"
format_stamp: "Formatted at 2026-08-08 04:07:11 on dist-test-slave-1l3l"
server_key: "1c35a255f052a90bdb28ea785819b122"
server_key_iv: "798b3c5a5fe4e2a133f9d9ba66eb5ecd"
server_key_version: "encryptionkey@0"
I20260808 04:07:11.937145 25656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/master-0/wal/instance:
uuid: "231726dbe3ec4fd49e58e8c2a800ed19"
format_stamp: "Formatted at 2026-08-08 04:07:11 on dist-test-slave-1l3l"
server_key: "1c35a255f052a90bdb28ea785819b122"
server_key_iv: "798b3c5a5fe4e2a133f9d9ba66eb5ecd"
server_key_version: "encryptionkey@0"
I20260808 04:07:11.941959 25656 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 04:07:11.945106 25680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:07:11.946255 25656 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 04:07:11.946421 25656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/master-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/master-0/wal
uuid: "231726dbe3ec4fd49e58e8c2a800ed19"
format_stamp: "Formatted at 2026-08-08 04:07:11 on dist-test-slave-1l3l"
server_key: "1c35a255f052a90bdb28ea785819b122"
server_key_iv: "798b3c5a5fe4e2a133f9d9ba66eb5ecd"
server_key_version: "encryptionkey@0"
I20260808 04:07:11.946586 25656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:07:11.987087 25656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:07:11.990901 25656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:07:11.991160 25656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:07:12.008055 25656 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.62:35721
I20260808 04:07:12.008100 25762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.62:35721 every 8 connection(s)
I20260808 04:07:12.009294 25656 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
I20260808 04:07:12.012734 25763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:07:12.013530 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 25656
I20260808 04:07:12.013728 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/master-0/wal/instance
I20260808 04:07:12.014350 5452 external_mini_cluster.cc:1428] Setting key 361f887fda788321f102c05272339b08
I20260808 04:07:12.026288 25763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 231726dbe3ec4fd49e58e8c2a800ed19: Bootstrap starting.
Aug 08 04:07:12 dist-test-slave-1l3l krb5kdc[25636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:07:12.029496 25763 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 231726dbe3ec4fd49e58e8c2a800ed19: Neither blocks nor log segments found. Creating new log.
I20260808 04:07:12.030539 25763 log.cc:824] T 00000000000000000000000000000000 P 231726dbe3ec4fd49e58e8c2a800ed19: Log is configured to *not* fsync() on all Append() calls
I20260808 04:07:12.033355 25763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 231726dbe3ec4fd49e58e8c2a800ed19: No bootstrap required, opened a new log
I20260808 04:07:12.036710 25767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:12.016304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33330 (local address 127.5.83.62:35721)
0808 04:07:12.017089 (+ 785us) server_negotiation.cc:206] Beginning negotiation
0808 04:07:12.017104 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 04:07:12.017155 (+ 51us) server_negotiation.cc:407] Connection header received
0808 04:07:12.018927 (+ 1772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:07:12.018953 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:07:12.019409 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:07:12.019855 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:07:12.021102 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:12.023016 (+ 1914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:12.024409 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:12.025410 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:12.025579 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:12.029750 (+ 4171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:07:12.029758 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:07:12.029770 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:07:12.029808 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:07:12.032835 (+ 3027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:12.033631 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:12.033637 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:12.033642 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:12.033803 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:12.034210 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:12.034213 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:12.034215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:12.034681 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:07:12.034901 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0808 04:07:12.035350 (+ 449us) server_negotiation.cc:299] Negotiation successful
0808 04:07:12.035673 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":318,"threads_started":1}
I20260808 04:07:12.041131 25763 raft_consensus.cc:348] T 00000000000000000000000000000000 P 231726dbe3ec4fd49e58e8c2a800ed19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "231726dbe3ec4fd49e58e8c2a800ed19" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 35721 } }
I20260808 04:07:12.041425 25763 raft_consensus.cc:374] T 00000000000000000000000000000000 P 231726dbe3ec4fd49e58e8c2a800ed19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:07:12.041512 25763 raft_consensus.cc:729] T 00000000000000000000000000000000 P 231726dbe3ec4fd49e58e8c2a800ed19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 231726dbe3ec4fd49e58e8c2a800ed19, State: Initialized, Role: FOLLOWER
I20260808 04:07:12.041986 25763 consensus_queue.cc:260] T 00000000000000000000000000000000 P 231726dbe3ec4fd49e58e8c2a800ed19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "231726dbe3ec4fd49e58e8c2a800ed19" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 35721 } }
I20260808 04:07:12.042184 25763 raft_consensus.cc:388] T 00000000000000000000000000000000 P 231726dbe3ec4fd49e58e8c2a800ed19 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:07:12.042303 25763 raft_consensus.cc:482] T 00000000000000000000000000000000 P 231726dbe3ec4fd49e58e8c2a800ed19 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:07:12.042452 25763 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 231726dbe3ec4fd49e58e8c2a800ed19 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:07:12.044428 25763 raft_consensus.cc:504] T 00000000000000000000000000000000 P 231726dbe3ec4fd49e58e8c2a800ed19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "231726dbe3ec4fd49e58e8c2a800ed19" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 35721 } }
I20260808 04:07:12.044842 25763 leader_election.cc:302] T 00000000000000000000000000000000 P 231726dbe3ec4fd49e58e8c2a800ed19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 231726dbe3ec4fd49e58e8c2a800ed19; no voters:
I20260808 04:07:12.045353 25763 leader_election.cc:288] T 00000000000000000000000000000000 P 231726dbe3ec4fd49e58e8c2a800ed19 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:07:12.045465 25771 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 231726dbe3ec4fd49e58e8c2a800ed19 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:07:12.046072 25771 raft_consensus.cc:686] T 00000000000000000000000000000000 P 231726dbe3ec4fd49e58e8c2a800ed19 [term 1 LEADER]: Becoming Leader. State: Replica: 231726dbe3ec4fd49e58e8c2a800ed19, State: Running, Role: LEADER
I20260808 04:07:12.046540 25771 consensus_queue.cc:237] T 00000000000000000000000000000000 P 231726dbe3ec4fd49e58e8c2a800ed19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "231726dbe3ec4fd49e58e8c2a800ed19" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 35721 } }
I20260808 04:07:12.047267 25763 sys_catalog.cc:564] T 00000000000000000000000000000000 P 231726dbe3ec4fd49e58e8c2a800ed19 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:07:12.049276 25772 sys_catalog.cc:455] T 00000000000000000000000000000000 P 231726dbe3ec4fd49e58e8c2a800ed19 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "231726dbe3ec4fd49e58e8c2a800ed19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "231726dbe3ec4fd49e58e8c2a800ed19" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 35721 } } }
I20260808 04:07:12.049765 25772 sys_catalog.cc:458] T 00000000000000000000000000000000 P 231726dbe3ec4fd49e58e8c2a800ed19 [sys.catalog]: This master's current role is: LEADER
I20260808 04:07:12.049300 25773 sys_catalog.cc:455] T 00000000000000000000000000000000 P 231726dbe3ec4fd49e58e8c2a800ed19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 231726dbe3ec4fd49e58e8c2a800ed19. Latest consensus state: current_term: 1 leader_uuid: "231726dbe3ec4fd49e58e8c2a800ed19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "231726dbe3ec4fd49e58e8c2a800ed19" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 35721 } } }
I20260808 04:07:12.050925 25773 sys_catalog.cc:458] T 00000000000000000000000000000000 P 231726dbe3ec4fd49e58e8c2a800ed19 [sys.catalog]: This master's current role is: LEADER
I20260808 04:07:12.051025 25780 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:07:12.055622 25780 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:07:12.064567 25780 catalog_manager.cc:1347] Generated new cluster ID: f2cf49dc4abb44b39ec9cf72bb35b00e
I20260808 04:07:12.064667 25780 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:07:12.106276 25780 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:07:12.108170 25780 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:07:12.115602 25780 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 231726dbe3ec4fd49e58e8c2a800ed19: Generated new TSK 0
I20260808 04:07:12.116542 25780 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.83.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.1.keytab.
Entry for principal kudu/127.5.83.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.1.keytab.
I20260808 04:07:12.180800 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.1:0
--local_ip_for_outbound_sockets=127.5.83.1
--webserver_interface=127.5.83.1
--webserver_port=0
--tserver_master_addrs=127.5.83.62:35721
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.1.keytab
--principal=kudu/127.5.83.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:39647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:07:12.497991 25806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:07:12.498245 25806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:07:12.498335 25806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:07:12.505309 25806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:07:12.505419 25806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:07:12.505548 25806 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.1
I20260808 04:07:12.514863 25806 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:39647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.1.keytab
--principal=kudu/127.5.83.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.83.1
--webserver_port=0
--tserver_master_addrs=127.5.83.62:35721
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.1
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:07:12.516072 25806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:07:12.517376 25806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:07:12.529458 25813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:07:12.529541 25814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:07:12.529564 25818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:07:12.530534 25806 server_base.cc:1034] running on GCE node
I20260808 04:07:12.531035 25806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:07:12.531872 25806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:07:12.533145 25806 hybrid_clock.cc:648] HybridClock initialized: now 1786162032533097 us; error 44 us; skew 500 ppm
I20260808 04:07:12.533430 25806 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:07:12 dist-test-slave-1l3l krb5kdc[25636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162032, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:07:12.536363 25806 init.cc:377] Logged in from keytab as kudu/127.5.83.1@KRBTEST.COM (short username kudu)
I20260808 04:07:12.540412 25806 webserver.cc:460] Webserver started at http://127.5.83.1:37729/ using document root <none> and password file <none>
I20260808 04:07:12.541081 25806 fs_manager.cc:359] Metadata directory not provided
I20260808 04:07:12.541198 25806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:07:12.541474 25806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:07:12.544271 25806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-0/data/instance:
uuid: "87ca4ca943204f63864c0587ad68d5e4"
format_stamp: "Formatted at 2026-08-08 04:07:12 on dist-test-slave-1l3l"
server_key: "6003d2fd8dc22147facf5cada69e4412"
server_key_iv: "2193ebbc6e935dc13dc2eb0474e0a27f"
server_key_version: "encryptionkey@0"
I20260808 04:07:12.544978 25806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-0/wal/instance:
uuid: "87ca4ca943204f63864c0587ad68d5e4"
format_stamp: "Formatted at 2026-08-08 04:07:12 on dist-test-slave-1l3l"
server_key: "6003d2fd8dc22147facf5cada69e4412"
server_key_iv: "2193ebbc6e935dc13dc2eb0474e0a27f"
server_key_version: "encryptionkey@0"
I20260808 04:07:12.549919 25806 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 04:07:12.553333 25828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:07:12.554607 25806 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 04:07:12.554754 25806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-0/wal
uuid: "87ca4ca943204f63864c0587ad68d5e4"
format_stamp: "Formatted at 2026-08-08 04:07:12 on dist-test-slave-1l3l"
server_key: "6003d2fd8dc22147facf5cada69e4412"
server_key_iv: "2193ebbc6e935dc13dc2eb0474e0a27f"
server_key_version: "encryptionkey@0"
I20260808 04:07:12.554924 25806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:07:12.573902 25806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:07:12.577782 25806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:07:12.578034 25806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:07:12.578912 25806 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:07:12.580212 25806 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:07:12.580291 25806 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:07:12.580417 25806 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:07:12.580472 25806 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:07:12.610811 25806 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.1:36007
I20260808 04:07:12.610852 25984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.1:36007 every 8 connection(s)
I20260808 04:07:12.612023 25806 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
I20260808 04:07:12.613320 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 25806
I20260808 04:07:12.613531 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-0/wal/instance
I20260808 04:07:12.614341 5452 external_mini_cluster.cc:1428] Setting key 4a29f8d7a7e80b6dd0e576878cb46e38
Aug 08 04:07:12 dist-test-slave-1l3l krb5kdc[25636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162032, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.1@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:07:12.631966 25990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:12.614590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.1:51551 (local address 127.5.83.62:35721)
0808 04:07:12.615107 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0808 04:07:12.615113 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:07:12.616174 (+ 1061us) server_negotiation.cc:407] Connection header received
0808 04:07:12.617958 (+ 1784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:07:12.617964 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:07:12.618068 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:07:12.618165 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:07:12.619584 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:12.620330 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:12.621898 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:12.622318 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:12.622439 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:12.626105 (+ 3666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:07:12.626111 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:07:12.626116 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:07:12.626156 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:07:12.628895 (+ 2739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:12.629779 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:12.629782 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:12.629785 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:12.629899 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:12.630328 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:12.630331 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:12.630333 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:12.630592 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:07:12.630700 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0808 04:07:12.631500 (+ 800us) server_negotiation.cc:299] Negotiation successful
0808 04:07:12.631669 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":226,"threads_started":1}
I20260808 04:07:12.633315 25991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:12.615195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:35721 (local address 127.5.83.1:51551)
0808 04:07:12.616030 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:12.616081 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 04:07:12.617704 (+ 1623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:07:12.618402 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:07:12.618410 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:07:12.618877 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:07:12.619419 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:07:12.619435 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:07:12.620482 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:07:12.620486 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:07:12.621744 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:07:12.621756 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:07:12.622538 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:07:12.622545 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:07:12.622754 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:12.623380 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:07:12.623407 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:07:12.625846 (+ 2439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:07:12.629040 (+ 3194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:07:12.629047 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:07:12.629062 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:07:12.629678 (+ 616us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:07:12.629999 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:07:12.630004 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:07:12.630007 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:07:12.630237 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:07:12.630718 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:07:12.630725 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:07:12.631121 (+ 396us) client_negotiation.cc:769] Sending connection context
0808 04:07:12.631406 (+ 285us) client_negotiation.cc:240] Negotiation successful
0808 04:07:12.631948 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":326,"threads_started":1}
I20260808 04:07:12.634908 25985 heartbeater.cc:347] Connected to a master server at 127.5.83.62:35721
I20260808 04:07:12.635258 25985 heartbeater.cc:464] Registering TS with master...
I20260808 04:07:12.636132 25985 heartbeater.cc:511] Master 127.5.83.62:35721 requested a full tablet report, sending...
I20260808 04:07:12.638290 25709 ts_manager.cc:194] Registered new tserver with Master: 87ca4ca943204f63864c0587ad68d5e4 (127.5.83.1:36007)
I20260808 04:07:12.639787 25709 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.1@KRBTEST.COM'} at 127.5.83.1:51551
WARNING: no policy specified for kudu/127.5.83.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.2.keytab.
Entry for principal kudu/127.5.83.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.2.keytab.
I20260808 04:07:12.674665 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.2:0
--local_ip_for_outbound_sockets=127.5.83.2
--webserver_interface=127.5.83.2
--webserver_port=0
--tserver_master_addrs=127.5.83.62:35721
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.2.keytab
--principal=kudu/127.5.83.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:39647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:07:12.998831 25997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:07:12.999086 25997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:07:12.999208 25997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:07:13.006245 25997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:07:13.006346 25997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:07:13.006505 25997 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.2
I20260808 04:07:13.015868 25997 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:39647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.2.keytab
--principal=kudu/127.5.83.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.83.2
--webserver_port=0
--tserver_master_addrs=127.5.83.62:35721
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.2
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:07:13.017020 25997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:07:13.018244 25997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:07:13.032428 26008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:07:13.032778 26013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:07:13.033866 25997 server_base.cc:1034] running on GCE node
W20260808 04:07:13.033898 26011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:07:13.034541 25997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:07:13.035455 25997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:07:13.036671 25997 hybrid_clock.cc:648] HybridClock initialized: now 1786162033036629 us; error 29 us; skew 500 ppm
I20260808 04:07:13.036975 25997 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:07:13 dist-test-slave-1l3l krb5kdc[25636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162033, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:07:13.040109 25997 init.cc:377] Logged in from keytab as kudu/127.5.83.2@KRBTEST.COM (short username kudu)
I20260808 04:07:13.041532 25997 webserver.cc:460] Webserver started at http://127.5.83.2:41049/ using document root <none> and password file <none>
I20260808 04:07:13.042191 25997 fs_manager.cc:359] Metadata directory not provided
I20260808 04:07:13.042294 25997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:07:13.042548 25997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:07:13.045358 25997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-1/data/instance:
uuid: "37ad64cb60b94c699ebb75ac30c3865c"
format_stamp: "Formatted at 2026-08-08 04:07:13 on dist-test-slave-1l3l"
server_key: "04f99e3d10123a3a0e3abe6a15c18711"
server_key_iv: "423165bcd7fceddee3de5393e2d68096"
server_key_version: "encryptionkey@0"
I20260808 04:07:13.046073 25997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-1/wal/instance:
uuid: "37ad64cb60b94c699ebb75ac30c3865c"
format_stamp: "Formatted at 2026-08-08 04:07:13 on dist-test-slave-1l3l"
server_key: "04f99e3d10123a3a0e3abe6a15c18711"
server_key_iv: "423165bcd7fceddee3de5393e2d68096"
server_key_version: "encryptionkey@0"
I20260808 04:07:13.051019 25997 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 04:07:13.054364 26024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:07:13.055583 25997 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 04:07:13.055751 25997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-1/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-1/wal
uuid: "37ad64cb60b94c699ebb75ac30c3865c"
format_stamp: "Formatted at 2026-08-08 04:07:13 on dist-test-slave-1l3l"
server_key: "04f99e3d10123a3a0e3abe6a15c18711"
server_key_iv: "423165bcd7fceddee3de5393e2d68096"
server_key_version: "encryptionkey@0"
I20260808 04:07:13.055925 25997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:07:13.066075 25997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:07:13.069962 25997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:07:13.070190 25997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:07:13.071003 25997 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:07:13.072234 25997 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:07:13.072307 25997 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:07:13.072414 25997 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:07:13.072458 25997 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:07:13.102481 25997 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.2:45521
I20260808 04:07:13.102521 26192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.2:45521 every 8 connection(s)
I20260808 04:07:13.103717 25997 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
I20260808 04:07:13.107239 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 25997
I20260808 04:07:13.107427 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-1/wal/instance
I20260808 04:07:13.108111 5452 external_mini_cluster.cc:1428] Setting key 2ed3b4173a381010241094403febad3b
Aug 08 04:07:13 dist-test-slave-1l3l krb5kdc[25636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162033, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.2@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:07:13.124589 25990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:13.106178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.2:33489 (local address 127.5.83.62:35721)
0808 04:07:13.106366 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0808 04:07:13.106371 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:07:13.107646 (+ 1275us) server_negotiation.cc:407] Connection header received
0808 04:07:13.109857 (+ 2211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:07:13.109863 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:07:13.109966 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:07:13.110082 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:07:13.111991 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:13.112709 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:13.114780 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:13.115251 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:13.115365 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:13.119131 (+ 3766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:07:13.119136 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:07:13.119139 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:07:13.119170 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:07:13.121669 (+ 2499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:13.122371 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:13.122375 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:13.122377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:13.122508 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:13.122916 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:13.122920 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:13.122922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:13.123189 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:07:13.123332 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0808 04:07:13.124026 (+ 694us) server_negotiation.cc:299] Negotiation successful
0808 04:07:13.124249 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 04:07:13.125478 26196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:13.106579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:35721 (local address 127.5.83.2:33489)
0808 04:07:13.107453 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:13.107512 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0808 04:07:13.109598 (+ 2086us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:07:13.110340 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:07:13.110351 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:07:13.111048 (+ 697us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:07:13.111811 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:07:13.111838 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:07:13.112872 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:07:13.112877 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:07:13.114625 (+ 1748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:07:13.114646 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:07:13.115471 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:07:13.115478 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:07:13.115779 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:13.116453 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:07:13.116481 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:07:13.118929 (+ 2448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:07:13.121812 (+ 2883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:07:13.121821 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:07:13.121835 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:07:13.122243 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:07:13.122614 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:07:13.122619 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:07:13.122622 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:07:13.122799 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:07:13.123332 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:07:13.123340 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:07:13.123631 (+ 291us) client_negotiation.cc:769] Sending connection context
0808 04:07:13.123906 (+ 275us) client_negotiation.cc:240] Negotiation successful
0808 04:07:13.124395 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":306,"threads_started":1}
I20260808 04:07:13.126768 26193 heartbeater.cc:347] Connected to a master server at 127.5.83.62:35721
I20260808 04:07:13.127099 26193 heartbeater.cc:464] Registering TS with master...
I20260808 04:07:13.127935 26193 heartbeater.cc:511] Master 127.5.83.62:35721 requested a full tablet report, sending...
I20260808 04:07:13.129415 25709 ts_manager.cc:194] Registered new tserver with Master: 37ad64cb60b94c699ebb75ac30c3865c (127.5.83.2:45521)
I20260808 04:07:13.130173 25709 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.2@KRBTEST.COM'} at 127.5.83.2:33489
WARNING: no policy specified for kudu/127.5.83.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.3.keytab.
Entry for principal kudu/127.5.83.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.3.keytab.
I20260808 04:07:13.164722 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.3:0
--local_ip_for_outbound_sockets=127.5.83.3
--webserver_interface=127.5.83.3
--webserver_port=0
--tserver_master_addrs=127.5.83.62:35721
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.3.keytab
--principal=kudu/127.5.83.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:39647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:07:13.477530 26202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:07:13.477779 26202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:07:13.477892 26202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:07:13.484923 26202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:07:13.485018 26202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:07:13.485158 26202 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.3
I20260808 04:07:13.494311 26202 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:39647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.3.keytab
--principal=kudu/127.5.83.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.83.3
--webserver_port=0
--tserver_master_addrs=127.5.83.62:35721
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.3
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:07:13.495504 26202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:07:13.496901 26202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:07:13.508987 26213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:07:13.509034 26216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:07:13.509063 26214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:07:13.509912 26202 server_base.cc:1034] running on GCE node
I20260808 04:07:13.510496 26202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:07:13.511319 26202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:07:13.512518 26202 hybrid_clock.cc:648] HybridClock initialized: now 1786162033512463 us; error 51 us; skew 500 ppm
I20260808 04:07:13.512804 26202 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:07:13 dist-test-slave-1l3l krb5kdc[25636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162033, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:07:13.515793 26202 init.cc:377] Logged in from keytab as kudu/127.5.83.3@KRBTEST.COM (short username kudu)
I20260808 04:07:13.517120 26202 webserver.cc:460] Webserver started at http://127.5.83.3:46075/ using document root <none> and password file <none>
I20260808 04:07:13.517750 26202 fs_manager.cc:359] Metadata directory not provided
I20260808 04:07:13.517867 26202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:07:13.518143 26202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:07:13.520958 26202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-2/data/instance:
uuid: "d9f897aa131b4d91a4cde42bef14dc48"
format_stamp: "Formatted at 2026-08-08 04:07:13 on dist-test-slave-1l3l"
server_key: "917e5a9e738808ecfa06cc7e9cbe0ddc"
server_key_iv: "ba4ec26d2ac5cec225a0c6ec200e4d02"
server_key_version: "encryptionkey@0"
I20260808 04:07:13.521669 26202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-2/wal/instance:
uuid: "d9f897aa131b4d91a4cde42bef14dc48"
format_stamp: "Formatted at 2026-08-08 04:07:13 on dist-test-slave-1l3l"
server_key: "917e5a9e738808ecfa06cc7e9cbe0ddc"
server_key_iv: "ba4ec26d2ac5cec225a0c6ec200e4d02"
server_key_version: "encryptionkey@0"
I20260808 04:07:13.526504 26202 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 04:07:13.529657 26227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:07:13.530874 26202 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:07:13.531013 26202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-2/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-2/wal
uuid: "d9f897aa131b4d91a4cde42bef14dc48"
format_stamp: "Formatted at 2026-08-08 04:07:13 on dist-test-slave-1l3l"
server_key: "917e5a9e738808ecfa06cc7e9cbe0ddc"
server_key_iv: "ba4ec26d2ac5cec225a0c6ec200e4d02"
server_key_version: "encryptionkey@0"
I20260808 04:07:13.531177 26202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:07:13.544550 26202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:07:13.548441 26202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:07:13.548683 26202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:07:13.549518 26202 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:07:13.550769 26202 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:07:13.550845 26202 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:07:13.550959 26202 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:07:13.551024 26202 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:07:13.580632 26202 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.3:36921
I20260808 04:07:13.580672 26385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.3:36921 every 8 connection(s)
I20260808 04:07:13.581813 26202 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
I20260808 04:07:13.585178 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 26202
I20260808 04:07:13.585377 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-2/wal/instance
I20260808 04:07:13.585978 5452 external_mini_cluster.cc:1428] Setting key bb5470b459a222c6d02ce654b69427f6
Aug 08 04:07:13 dist-test-slave-1l3l krb5kdc[25636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162033, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.3@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:07:13.600782 25990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:13.584195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.3:59099 (local address 127.5.83.62:35721)
0808 04:07:13.584371 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0808 04:07:13.584376 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:07:13.585635 (+ 1259us) server_negotiation.cc:407] Connection header received
0808 04:07:13.587821 (+ 2186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:07:13.587827 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:07:13.587912 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:07:13.587992 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:07:13.589562 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:13.590312 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:13.591829 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:13.592221 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:13.592312 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:13.595436 (+ 3124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:07:13.595441 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:07:13.595444 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:07:13.595476 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:07:13.597926 (+ 2450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:13.598527 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:13.598531 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:13.598533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:13.598652 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:13.599063 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:13.599066 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:13.599068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:13.599317 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:07:13.599461 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0808 04:07:13.600263 (+ 802us) server_negotiation.cc:299] Negotiation successful
0808 04:07:13.600462 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 04:07:13.601682 26391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:13.584627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:35721 (local address 127.5.83.3:59099)
0808 04:07:13.585463 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:13.585519 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0808 04:07:13.587487 (+ 1968us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:07:13.588237 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:07:13.588245 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:07:13.588791 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:07:13.589407 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:07:13.589424 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:07:13.590440 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:07:13.590444 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:07:13.591657 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:07:13.591669 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:07:13.592388 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:07:13.592392 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:07:13.592592 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:13.593153 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:07:13.593174 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:07:13.595226 (+ 2052us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:07:13.598050 (+ 2824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:07:13.598056 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:07:13.598067 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:07:13.598436 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:07:13.598771 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:07:13.598774 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:07:13.598777 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:07:13.598972 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:07:13.599471 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:07:13.599477 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:07:13.599859 (+ 382us) client_negotiation.cc:769] Sending connection context
0808 04:07:13.600143 (+ 284us) client_negotiation.cc:240] Negotiation successful
0808 04:07:13.600552 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":328,"threads_started":1}
I20260808 04:07:13.602818 26386 heartbeater.cc:347] Connected to a master server at 127.5.83.62:35721
I20260808 04:07:13.603062 26386 heartbeater.cc:464] Registering TS with master...
I20260808 04:07:13.603683 26386 heartbeater.cc:511] Master 127.5.83.62:35721 requested a full tablet report, sending...
I20260808 04:07:13.605014 25709 ts_manager.cc:194] Registered new tserver with Master: d9f897aa131b4d91a4cde42bef14dc48 (127.5.83.3:36921)
I20260808 04:07:13.605837 25709 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.3@KRBTEST.COM'} at 127.5.83.3:59099
I20260808 04:07:13.613200 5452 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:07:13.629662 25990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:13.617441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33346 (local address 127.5.83.62:35721)
0808 04:07:13.617579 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0808 04:07:13.617583 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:07:13.618059 (+ 476us) server_negotiation.cc:407] Connection header received
0808 04:07:13.618185 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:07:13.618188 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:07:13.618244 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:07:13.618320 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:07:13.619680 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:13.620447 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:13.621709 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:13.622117 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:13.622207 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:13.624585 (+ 2378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:07:13.624592 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:07:13.624596 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:07:13.624630 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:07:13.627281 (+ 2651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:13.627951 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:13.627955 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:13.627957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:13.628079 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:13.628410 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:13.628414 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:13.628416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:13.628728 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:07:13.628941 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0808 04:07:13.629245 (+ 304us) server_negotiation.cc:299] Negotiation successful
0808 04:07:13.629436 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 04:07:13.633263 25709 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33346:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:07:13.639977 25709 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:07:13.643111 25985 heartbeater.cc:503] Master 127.5.83.62:35721 was elected leader, sending a full tablet report...
I20260808 04:07:13.664011 26406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:13.652345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.2:45521 (local address 127.0.0.1:54012)
0808 04:07:13.653663 (+ 1318us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:13.653703 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 04:07:13.653873 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:07:13.654817 (+ 944us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:07:13.654825 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:07:13.654856 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:07:13.655253 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:07:13.655266 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:07:13.657833 (+ 2567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:07:13.657839 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:07:13.660739 (+ 2900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:07:13.660754 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:07:13.663004 (+ 2250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:07:13.663011 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:07:13.663247 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:13.663297 (+ 50us) client_negotiation.cc:769] Sending connection context
0808 04:07:13.663438 (+ 141us) client_negotiation.cc:240] Negotiation successful
0808 04:07:13.663629 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1067,"thread_start_us":916,"threads_started":1}
I20260808 04:07:13.664011 26408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:13.652586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54012 (local address 127.5.83.2:45521)
0808 04:07:13.653143 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0808 04:07:13.653152 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 04:07:13.653815 (+ 663us) server_negotiation.cc:407] Connection header received
0808 04:07:13.654093 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:07:13.654101 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:07:13.654346 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:07:13.654542 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:07:13.655468 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:13.657698 (+ 2230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:13.660916 (+ 3218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:13.662466 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:13.662583 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:13.662794 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0808 04:07:13.663496 (+ 702us) server_negotiation.cc:299] Negotiation successful
0808 04:07:13.663664 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":347,"threads_started":1}
I20260808 04:07:13.668239 26099 tablet_service.cc:1467] Processing CreateTablet for tablet ed45ce5bd6424a5aa2402d83be3c1c97 (DEFAULT_TABLE table=test-table [id=1bb1f78c0c2d479db4288a884812d1d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:07:13.669581 26099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed45ce5bd6424a5aa2402d83be3c1c97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:07:13.669617 26410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:13.653045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39734 (local address 127.5.83.1:36007)
0808 04:07:13.661601 (+ 8556us) server_negotiation.cc:206] Beginning negotiation
0808 04:07:13.661610 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 04:07:13.661640 (+ 30us) server_negotiation.cc:407] Connection header received
0808 04:07:13.661738 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:07:13.661746 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:07:13.661981 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:07:13.662174 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:07:13.663801 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:13.665504 (+ 1703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:13.667321 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:13.668292 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:13.668415 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:13.668650 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0808 04:07:13.669158 (+ 508us) server_negotiation.cc:299] Negotiation successful
0808 04:07:13.669319 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8362,"thread_start_us":8187,"threads_started":1}
I20260808 04:07:13.669917 26411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:13.652490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:36007 (local address 127.0.0.1:39734)
0808 04:07:13.660236 (+ 7746us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:13.660305 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0808 04:07:13.660429 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:07:13.662418 (+ 1989us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:07:13.662423 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:07:13.662452 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:07:13.662902 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:07:13.662913 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:07:13.665628 (+ 2715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:07:13.665633 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:07:13.667173 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:07:13.667184 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:07:13.668792 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:07:13.668798 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:07:13.668964 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:13.668997 (+ 33us) client_negotiation.cc:769] Sending connection context
0808 04:07:13.669097 (+ 100us) client_negotiation.cc:240] Negotiation successful
0808 04:07:13.669220 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7597,"thread_start_us":10719,"threads_started":1}
I20260808 04:07:13.673116 26407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:13.653576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.3:36921 (local address 127.0.0.1:51878)
0808 04:07:13.663487 (+ 9911us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:13.663505 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 04:07:13.663606 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:07:13.666081 (+ 2475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:07:13.666086 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:07:13.666112 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:07:13.666485 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:07:13.666502 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:07:13.668331 (+ 1829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:07:13.668336 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:07:13.670667 (+ 2331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:07:13.670679 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:07:13.672391 (+ 1712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:07:13.672397 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:07:13.672566 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:13.672598 (+ 32us) client_negotiation.cc:769] Sending connection context
0808 04:07:13.672691 (+ 93us) client_negotiation.cc:240] Negotiation successful
0808 04:07:13.672803 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9778,"thread_start_us":6296,"threads_started":1}
I20260808 04:07:13.673240 26412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:13.654066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51878 (local address 127.5.83.3:36921)
0808 04:07:13.665519 (+ 11453us) server_negotiation.cc:206] Beginning negotiation
0808 04:07:13.665527 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:07:13.665561 (+ 34us) server_negotiation.cc:407] Connection header received
0808 04:07:13.665659 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:07:13.665665 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:07:13.665876 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:07:13.666080 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:07:13.666670 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:13.668182 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:13.670814 (+ 2632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:13.672185 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:13.672320 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:13.672527 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0808 04:07:13.672751 (+ 224us) server_negotiation.cc:299] Negotiation successful
0808 04:07:13.672899 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10490,"thread_start_us":10324,"threads_started":1}
I20260808 04:07:13.673581 25898 tablet_service.cc:1467] Processing CreateTablet for tablet ed45ce5bd6424a5aa2402d83be3c1c97 (DEFAULT_TABLE table=test-table [id=1bb1f78c0c2d479db4288a884812d1d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:07:13.674819 25898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed45ce5bd6424a5aa2402d83be3c1c97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:07:13.676743 26287 tablet_service.cc:1467] Processing CreateTablet for tablet ed45ce5bd6424a5aa2402d83be3c1c97 (DEFAULT_TABLE table=test-table [id=1bb1f78c0c2d479db4288a884812d1d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:07:13.678061 26287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed45ce5bd6424a5aa2402d83be3c1c97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:07:13.683776 26414 tablet_bootstrap.cc:492] T ed45ce5bd6424a5aa2402d83be3c1c97 P 37ad64cb60b94c699ebb75ac30c3865c: Bootstrap starting.
I20260808 04:07:13.685869 26420 tablet_bootstrap.cc:492] T ed45ce5bd6424a5aa2402d83be3c1c97 P 87ca4ca943204f63864c0587ad68d5e4: Bootstrap starting.
I20260808 04:07:13.687189 26414 tablet_bootstrap.cc:654] T ed45ce5bd6424a5aa2402d83be3c1c97 P 37ad64cb60b94c699ebb75ac30c3865c: Neither blocks nor log segments found. Creating new log.
I20260808 04:07:13.688227 26414 log.cc:824] T ed45ce5bd6424a5aa2402d83be3c1c97 P 37ad64cb60b94c699ebb75ac30c3865c: Log is configured to *not* fsync() on all Append() calls
I20260808 04:07:13.689184 26420 tablet_bootstrap.cc:654] T ed45ce5bd6424a5aa2402d83be3c1c97 P 87ca4ca943204f63864c0587ad68d5e4: Neither blocks nor log segments found. Creating new log.
I20260808 04:07:13.690347 26420 log.cc:824] T ed45ce5bd6424a5aa2402d83be3c1c97 P 87ca4ca943204f63864c0587ad68d5e4: Log is configured to *not* fsync() on all Append() calls
I20260808 04:07:13.690791 26414 tablet_bootstrap.cc:492] T ed45ce5bd6424a5aa2402d83be3c1c97 P 37ad64cb60b94c699ebb75ac30c3865c: No bootstrap required, opened a new log
I20260808 04:07:13.691084 26414 ts_tablet_manager.cc:1397] T ed45ce5bd6424a5aa2402d83be3c1c97 P 37ad64cb60b94c699ebb75ac30c3865c: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260808 04:07:13.692378 26423 tablet_bootstrap.cc:492] T ed45ce5bd6424a5aa2402d83be3c1c97 P d9f897aa131b4d91a4cde42bef14dc48: Bootstrap starting.
I20260808 04:07:13.695626 26423 tablet_bootstrap.cc:654] T ed45ce5bd6424a5aa2402d83be3c1c97 P d9f897aa131b4d91a4cde42bef14dc48: Neither blocks nor log segments found. Creating new log.
I20260808 04:07:13.696753 26423 log.cc:824] T ed45ce5bd6424a5aa2402d83be3c1c97 P d9f897aa131b4d91a4cde42bef14dc48: Log is configured to *not* fsync() on all Append() calls
I20260808 04:07:13.697928 26420 tablet_bootstrap.cc:492] T ed45ce5bd6424a5aa2402d83be3c1c97 P 87ca4ca943204f63864c0587ad68d5e4: No bootstrap required, opened a new log
I20260808 04:07:13.698182 26420 ts_tablet_manager.cc:1397] T ed45ce5bd6424a5aa2402d83be3c1c97 P 87ca4ca943204f63864c0587ad68d5e4: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.007s
I20260808 04:07:13.699632 26423 tablet_bootstrap.cc:492] T ed45ce5bd6424a5aa2402d83be3c1c97 P d9f897aa131b4d91a4cde42bef14dc48: No bootstrap required, opened a new log
I20260808 04:07:13.699919 26423 ts_tablet_manager.cc:1397] T ed45ce5bd6424a5aa2402d83be3c1c97 P d9f897aa131b4d91a4cde42bef14dc48: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.002s
I20260808 04:07:13.699878 26414 raft_consensus.cc:348] T ed45ce5bd6424a5aa2402d83be3c1c97 P 37ad64cb60b94c699ebb75ac30c3865c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ad64cb60b94c699ebb75ac30c3865c" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 45521 } } peers { permanent_uuid: "87ca4ca943204f63864c0587ad68d5e4" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 36007 } } peers { permanent_uuid: "d9f897aa131b4d91a4cde42bef14dc48" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 36921 } }
I20260808 04:07:13.700239 26414 raft_consensus.cc:374] T ed45ce5bd6424a5aa2402d83be3c1c97 P 37ad64cb60b94c699ebb75ac30c3865c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:07:13.700338 26414 raft_consensus.cc:729] T ed45ce5bd6424a5aa2402d83be3c1c97 P 37ad64cb60b94c699ebb75ac30c3865c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37ad64cb60b94c699ebb75ac30c3865c, State: Initialized, Role: FOLLOWER
I20260808 04:07:13.700891 26414 consensus_queue.cc:260] T ed45ce5bd6424a5aa2402d83be3c1c97 P 37ad64cb60b94c699ebb75ac30c3865c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ad64cb60b94c699ebb75ac30c3865c" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 45521 } } peers { permanent_uuid: "87ca4ca943204f63864c0587ad68d5e4" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 36007 } } peers { permanent_uuid: "d9f897aa131b4d91a4cde42bef14dc48" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 36921 } }
I20260808 04:07:13.702865 26193 heartbeater.cc:503] Master 127.5.83.62:35721 was elected leader, sending a full tablet report...
I20260808 04:07:13.704428 26414 ts_tablet_manager.cc:1428] T ed45ce5bd6424a5aa2402d83be3c1c97 P 37ad64cb60b94c699ebb75ac30c3865c: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20260808 04:07:13.708176 26420 raft_consensus.cc:348] T ed45ce5bd6424a5aa2402d83be3c1c97 P 87ca4ca943204f63864c0587ad68d5e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ad64cb60b94c699ebb75ac30c3865c" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 45521 } } peers { permanent_uuid: "87ca4ca943204f63864c0587ad68d5e4" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 36007 } } peers { permanent_uuid: "d9f897aa131b4d91a4cde42bef14dc48" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 36921 } }
I20260808 04:07:13.708580 26420 raft_consensus.cc:374] T ed45ce5bd6424a5aa2402d83be3c1c97 P 87ca4ca943204f63864c0587ad68d5e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:07:13.708675 26420 raft_consensus.cc:729] T ed45ce5bd6424a5aa2402d83be3c1c97 P 87ca4ca943204f63864c0587ad68d5e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ca4ca943204f63864c0587ad68d5e4, State: Initialized, Role: FOLLOWER
I20260808 04:07:13.709225 26420 consensus_queue.cc:260] T ed45ce5bd6424a5aa2402d83be3c1c97 P 87ca4ca943204f63864c0587ad68d5e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ad64cb60b94c699ebb75ac30c3865c" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 45521 } } peers { permanent_uuid: "87ca4ca943204f63864c0587ad68d5e4" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 36007 } } peers { permanent_uuid: "d9f897aa131b4d91a4cde42bef14dc48" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 36921 } }
I20260808 04:07:13.709800 26423 raft_consensus.cc:348] T ed45ce5bd6424a5aa2402d83be3c1c97 P d9f897aa131b4d91a4cde42bef14dc48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ad64cb60b94c699ebb75ac30c3865c" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 45521 } } peers { permanent_uuid: "87ca4ca943204f63864c0587ad68d5e4" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 36007 } } peers { permanent_uuid: "d9f897aa131b4d91a4cde42bef14dc48" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 36921 } }
I20260808 04:07:13.710206 26423 raft_consensus.cc:374] T ed45ce5bd6424a5aa2402d83be3c1c97 P d9f897aa131b4d91a4cde42bef14dc48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:07:13.710304 26423 raft_consensus.cc:729] T ed45ce5bd6424a5aa2402d83be3c1c97 P d9f897aa131b4d91a4cde42bef14dc48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9f897aa131b4d91a4cde42bef14dc48, State: Initialized, Role: FOLLOWER
I20260808 04:07:13.710876 26420 ts_tablet_manager.cc:1428] T ed45ce5bd6424a5aa2402d83be3c1c97 P 87ca4ca943204f63864c0587ad68d5e4: Time spent starting tablet: real 0.013s user 0.008s sys 0.005s
I20260808 04:07:13.710973 26423 consensus_queue.cc:260] T ed45ce5bd6424a5aa2402d83be3c1c97 P d9f897aa131b4d91a4cde42bef14dc48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ad64cb60b94c699ebb75ac30c3865c" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 45521 } } peers { permanent_uuid: "87ca4ca943204f63864c0587ad68d5e4" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 36007 } } peers { permanent_uuid: "d9f897aa131b4d91a4cde42bef14dc48" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 36921 } }
I20260808 04:07:13.711995 26386 heartbeater.cc:503] Master 127.5.83.62:35721 was elected leader, sending a full tablet report...
I20260808 04:07:13.712499 26423 ts_tablet_manager.cc:1428] T ed45ce5bd6424a5aa2402d83be3c1c97 P d9f897aa131b4d91a4cde42bef14dc48: Time spent starting tablet: real 0.012s user 0.011s sys 0.001s
I20260808 04:07:13.741173 26425 raft_consensus.cc:482] T ed45ce5bd6424a5aa2402d83be3c1c97 P 37ad64cb60b94c699ebb75ac30c3865c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:07:13.741431 26425 raft_consensus.cc:504] T ed45ce5bd6424a5aa2402d83be3c1c97 P 37ad64cb60b94c699ebb75ac30c3865c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ad64cb60b94c699ebb75ac30c3865c" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 45521 } } peers { permanent_uuid: "87ca4ca943204f63864c0587ad68d5e4" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 36007 } } peers { permanent_uuid: "d9f897aa131b4d91a4cde42bef14dc48" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 36921 } }
I20260808 04:07:13.743031 26425 leader_election.cc:288] T ed45ce5bd6424a5aa2402d83be3c1c97 P 37ad64cb60b94c699ebb75ac30c3865c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ca4ca943204f63864c0587ad68d5e4 (127.5.83.1:36007), d9f897aa131b4d91a4cde42bef14dc48 (127.5.83.3:36921)
I20260808 04:07:13.751513 26410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:13.743411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.2:52471 (local address 127.5.83.1:36007)
0808 04:07:13.743601 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0808 04:07:13.743607 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:07:13.744360 (+ 753us) server_negotiation.cc:407] Connection header received
0808 04:07:13.744566 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:07:13.744571 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:07:13.744646 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:07:13.744738 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:07:13.745456 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:13.746582 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:13.749515 (+ 2933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:13.750537 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:13.750675 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:13.750754 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0808 04:07:13.751138 (+ 384us) server_negotiation.cc:299] Negotiation successful
0808 04:07:13.751264 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 04:07:13.751598 26412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:13.743500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.2:37473 (local address 127.5.83.3:36921)
0808 04:07:13.743667 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0808 04:07:13.743672 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:07:13.744453 (+ 781us) server_negotiation.cc:407] Connection header received
0808 04:07:13.744575 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:07:13.744580 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:07:13.744646 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:07:13.744742 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:07:13.745460 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:13.746575 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:13.749558 (+ 2983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:13.750491 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:13.750611 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:13.750715 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0808 04:07:13.751168 (+ 453us) server_negotiation.cc:299] Negotiation successful
0808 04:07:13.751376 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 04:07:13.751747 26428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:13.743460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.3:36921 (local address 127.5.83.2:37473)
0808 04:07:13.744352 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:13.744366 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:07:13.744474 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:07:13.744764 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:07:13.744769 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:07:13.744805 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:07:13.745285 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:07:13.745302 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:07:13.746711 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:07:13.746716 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:07:13.749363 (+ 2647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:07:13.749377 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:07:13.750848 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:07:13.750853 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:07:13.751031 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:13.751056 (+ 25us) client_negotiation.cc:769] Sending connection context
0808 04:07:13.751130 (+ 74us) client_negotiation.cc:240] Negotiation successful
0808 04:07:13.751263 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":779,"mutex_wait_us":206,"thread_start_us":331,"threads_started":1}
I20260808 04:07:13.751868 26429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:13.743251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:36007 (local address 127.5.83.2:52471)
0808 04:07:13.744242 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:13.744266 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0808 04:07:13.744420 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:07:13.744764 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:07:13.744769 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:07:13.744859 (+ 90us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:07:13.745316 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:07:13.745329 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:07:13.746714 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:07:13.746718 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:07:13.749362 (+ 2644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:07:13.749377 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:07:13.750750 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:07:13.750756 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:07:13.750978 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:13.751014 (+ 36us) client_negotiation.cc:769] Sending connection context
0808 04:07:13.751113 (+ 99us) client_negotiation.cc:240] Negotiation successful
0808 04:07:13.751263 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":833,"mutex_wait_us":124,"spinlock_wait_cycles":20736,"thread_start_us":681,"threads_started":1}
I20260808 04:07:13.753012 26315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed45ce5bd6424a5aa2402d83be3c1c97" candidate_uuid: "37ad64cb60b94c699ebb75ac30c3865c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9f897aa131b4d91a4cde42bef14dc48" is_pre_election: true
I20260808 04:07:13.753001 25928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed45ce5bd6424a5aa2402d83be3c1c97" candidate_uuid: "37ad64cb60b94c699ebb75ac30c3865c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ca4ca943204f63864c0587ad68d5e4" is_pre_election: true
I20260808 04:07:13.753557 25928 raft_consensus.cc:2454] T ed45ce5bd6424a5aa2402d83be3c1c97 P 87ca4ca943204f63864c0587ad68d5e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37ad64cb60b94c699ebb75ac30c3865c in term 0.
I20260808 04:07:13.753552 26315 raft_consensus.cc:2454] T ed45ce5bd6424a5aa2402d83be3c1c97 P d9f897aa131b4d91a4cde42bef14dc48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37ad64cb60b94c699ebb75ac30c3865c in term 0.
I20260808 04:07:13.754206 26028 leader_election.cc:302] T ed45ce5bd6424a5aa2402d83be3c1c97 P 37ad64cb60b94c699ebb75ac30c3865c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37ad64cb60b94c699ebb75ac30c3865c, d9f897aa131b4d91a4cde42bef14dc48; no voters:
I20260808 04:07:13.754590 26425 raft_consensus.cc:2781] T ed45ce5bd6424a5aa2402d83be3c1c97 P 37ad64cb60b94c699ebb75ac30c3865c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:07:13.754722 26425 raft_consensus.cc:482] T ed45ce5bd6424a5aa2402d83be3c1c97 P 37ad64cb60b94c699ebb75ac30c3865c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:07:13.754822 26425 raft_consensus.cc:3037] T ed45ce5bd6424a5aa2402d83be3c1c97 P 37ad64cb60b94c699ebb75ac30c3865c [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:07:13.757061 26425 raft_consensus.cc:504] T ed45ce5bd6424a5aa2402d83be3c1c97 P 37ad64cb60b94c699ebb75ac30c3865c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ad64cb60b94c699ebb75ac30c3865c" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 45521 } } peers { permanent_uuid: "87ca4ca943204f63864c0587ad68d5e4" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 36007 } } peers { permanent_uuid: "d9f897aa131b4d91a4cde42bef14dc48" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 36921 } }
I20260808 04:07:13.757581 26425 leader_election.cc:288] T ed45ce5bd6424a5aa2402d83be3c1c97 P 37ad64cb60b94c699ebb75ac30c3865c [CANDIDATE]: Term 1 election: Requested vote from peers 87ca4ca943204f63864c0587ad68d5e4 (127.5.83.1:36007), d9f897aa131b4d91a4cde42bef14dc48 (127.5.83.3:36921)
I20260808 04:07:13.757970 25928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed45ce5bd6424a5aa2402d83be3c1c97" candidate_uuid: "37ad64cb60b94c699ebb75ac30c3865c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ca4ca943204f63864c0587ad68d5e4"
I20260808 04:07:13.758002 26315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed45ce5bd6424a5aa2402d83be3c1c97" candidate_uuid: "37ad64cb60b94c699ebb75ac30c3865c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9f897aa131b4d91a4cde42bef14dc48"
I20260808 04:07:13.758144 25928 raft_consensus.cc:3037] T ed45ce5bd6424a5aa2402d83be3c1c97 P 87ca4ca943204f63864c0587ad68d5e4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:07:13.758177 26315 raft_consensus.cc:3037] T ed45ce5bd6424a5aa2402d83be3c1c97 P d9f897aa131b4d91a4cde42bef14dc48 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:07:13.760468 26315 raft_consensus.cc:2454] T ed45ce5bd6424a5aa2402d83be3c1c97 P d9f897aa131b4d91a4cde42bef14dc48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37ad64cb60b94c699ebb75ac30c3865c in term 1.
I20260808 04:07:13.760600 25928 raft_consensus.cc:2454] T ed45ce5bd6424a5aa2402d83be3c1c97 P 87ca4ca943204f63864c0587ad68d5e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37ad64cb60b94c699ebb75ac30c3865c in term 1.
I20260808 04:07:13.760828 26028 leader_election.cc:302] T ed45ce5bd6424a5aa2402d83be3c1c97 P 37ad64cb60b94c699ebb75ac30c3865c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37ad64cb60b94c699ebb75ac30c3865c, d9f897aa131b4d91a4cde42bef14dc48; no voters:
I20260808 04:07:13.761126 26425 raft_consensus.cc:2781] T ed45ce5bd6424a5aa2402d83be3c1c97 P 37ad64cb60b94c699ebb75ac30c3865c [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:07:13.761533 26425 raft_consensus.cc:686] T ed45ce5bd6424a5aa2402d83be3c1c97 P 37ad64cb60b94c699ebb75ac30c3865c [term 1 LEADER]: Becoming Leader. State: Replica: 37ad64cb60b94c699ebb75ac30c3865c, State: Running, Role: LEADER
I20260808 04:07:13.761993 26425 consensus_queue.cc:237] T ed45ce5bd6424a5aa2402d83be3c1c97 P 37ad64cb60b94c699ebb75ac30c3865c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ad64cb60b94c699ebb75ac30c3865c" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 45521 } } peers { permanent_uuid: "87ca4ca943204f63864c0587ad68d5e4" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 36007 } } peers { permanent_uuid: "d9f897aa131b4d91a4cde42bef14dc48" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 36921 } }
I20260808 04:07:13.766005 25706 catalog_manager.cc:5495] T ed45ce5bd6424a5aa2402d83be3c1c97 P 37ad64cb60b94c699ebb75ac30c3865c reported cstate change: term changed from 0 to 1, leader changed from <none> to 37ad64cb60b94c699ebb75ac30c3865c (127.5.83.2). New cstate: current_term: 1 leader_uuid: "37ad64cb60b94c699ebb75ac30c3865c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ad64cb60b94c699ebb75ac30c3865c" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 45521 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "87ca4ca943204f63864c0587ad68d5e4" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 36007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d9f897aa131b4d91a4cde42bef14dc48" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 36921 } health_report { overall_health: UNKNOWN } } }
I20260808 04:07:13.784684 26408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:13.779890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54022 (local address 127.5.83.2:45521)
0808 04:07:13.780099 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0808 04:07:13.780104 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:07:13.780142 (+ 38us) server_negotiation.cc:407] Connection header received
0808 04:07:13.780314 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:07:13.780317 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:07:13.780392 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:07:13.780509 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:07:13.780810 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:13.781510 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:13.782778 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:13.783166 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:13.783256 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:13.783689 (+ 433us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:07:13.784197 (+ 508us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:07:13.784300 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0808 04:07:13.784369 (+ 69us) server_negotiation.cc:299] Negotiation successful
0808 04:07:13.784430 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 04:07:13.791972 26315 raft_consensus.cc:1261] T ed45ce5bd6424a5aa2402d83be3c1c97 P d9f897aa131b4d91a4cde42bef14dc48 [term 1 FOLLOWER]: Refusing update from remote peer 37ad64cb60b94c699ebb75ac30c3865c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:07:13.792006 25928 raft_consensus.cc:1261] T ed45ce5bd6424a5aa2402d83be3c1c97 P 87ca4ca943204f63864c0587ad68d5e4 [term 1 FOLLOWER]: Refusing update from remote peer 37ad64cb60b94c699ebb75ac30c3865c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:07:13.792826 26430 consensus_queue.cc:1035] T ed45ce5bd6424a5aa2402d83be3c1c97 P 37ad64cb60b94c699ebb75ac30c3865c [LEADER]: Connected to new peer: Peer: permanent_uuid: "87ca4ca943204f63864c0587ad68d5e4" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 36007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:07:13.793165 26425 consensus_queue.cc:1035] T ed45ce5bd6424a5aa2402d83be3c1c97 P 37ad64cb60b94c699ebb75ac30c3865c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9f897aa131b4d91a4cde42bef14dc48" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 36921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:07:13.802584 26434 mvcc.cc:205] Tried to move back new op lower bound from 7316119690403184640 to 7316119690292633600. Current Snapshot: MvccSnapshot[applied={T|T < 7316119690403184640}]
I20260808 04:07:13.803360 26436 mvcc.cc:205] Tried to move back new op lower bound from 7316119690403184640 to 7316119690292633600. Current Snapshot: MvccSnapshot[applied={T|T < 7316119690403184640}]
I20260808 04:07:13.815613 25704 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33346:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:07:13.815917 25704 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33346:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:07:13.819204 25704 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 231726dbe3ec4fd49e58e8c2a800ed19: Sending DeleteTablet for 3 replicas of tablet ed45ce5bd6424a5aa2402d83be3c1c97
I20260808 04:07:13.820436 26099 tablet_service.cc:1514] Processing DeleteTablet for tablet ed45ce5bd6424a5aa2402d83be3c1c97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:07:13 UTC) from {username='kudu', principal='kudu/127.5.83.62@KRBTEST.COM'} at 127.0.0.1:54012
I20260808 04:07:13.820427 25898 tablet_service.cc:1514] Processing DeleteTablet for tablet ed45ce5bd6424a5aa2402d83be3c1c97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:07:13 UTC) from {username='kudu', principal='kudu/127.5.83.62@KRBTEST.COM'} at 127.0.0.1:39734
I20260808 04:07:13.821216 26451 tablet_replica.cc:330] T ed45ce5bd6424a5aa2402d83be3c1c97 P 37ad64cb60b94c699ebb75ac30c3865c: stopping tablet replica
I20260808 04:07:13.821246 26287 tablet_service.cc:1514] Processing DeleteTablet for tablet ed45ce5bd6424a5aa2402d83be3c1c97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:07:13 UTC) from {username='kudu', principal='kudu/127.5.83.62@KRBTEST.COM'} at 127.0.0.1:51878
I20260808 04:07:13.821666 26451 raft_consensus.cc:2229] T ed45ce5bd6424a5aa2402d83be3c1c97 P 37ad64cb60b94c699ebb75ac30c3865c [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:07:13.822351 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 25806
I20260808 04:07:13.822328 26451 raft_consensus.cc:2258] T ed45ce5bd6424a5aa2402d83be3c1c97 P 37ad64cb60b94c699ebb75ac30c3865c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:07:13.824466 26451 ts_tablet_manager.cc:1905] T ed45ce5bd6424a5aa2402d83be3c1c97 P 37ad64cb60b94c699ebb75ac30c3865c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:07:13.831995 26451 ts_tablet_manager.cc:1918] T ed45ce5bd6424a5aa2402d83be3c1c97 P 37ad64cb60b94c699ebb75ac30c3865c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:07:13.832226 26451 log.cc:1196] T ed45ce5bd6424a5aa2402d83be3c1c97 P 37ad64cb60b94c699ebb75ac30c3865c: Deleting WAL directory at /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-1/wal/wals/ed45ce5bd6424a5aa2402d83be3c1c97
I20260808 04:07:13.832825 26451 ts_tablet_manager.cc:1939] T ed45ce5bd6424a5aa2402d83be3c1c97 P 37ad64cb60b94c699ebb75ac30c3865c: Deleting consensus metadata
I20260808 04:07:13.834568 25685 catalog_manager.cc:4841] TS 37ad64cb60b94c699ebb75ac30c3865c (127.5.83.2:45521): tablet ed45ce5bd6424a5aa2402d83be3c1c97 (table test-table [id=1bb1f78c0c2d479db4288a884812d1d9]) successfully deleted
I20260808 04:07:13.836902 26453 tablet_replica.cc:330] T ed45ce5bd6424a5aa2402d83be3c1c97 P d9f897aa131b4d91a4cde42bef14dc48: stopping tablet replica
I20260808 04:07:13.837376 26453 raft_consensus.cc:2229] T ed45ce5bd6424a5aa2402d83be3c1c97 P d9f897aa131b4d91a4cde42bef14dc48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:07:13.837946 26453 raft_consensus.cc:2258] T ed45ce5bd6424a5aa2402d83be3c1c97 P d9f897aa131b4d91a4cde42bef14dc48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:07:13.840015 26453 ts_tablet_manager.cc:1905] T ed45ce5bd6424a5aa2402d83be3c1c97 P d9f897aa131b4d91a4cde42bef14dc48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:07:13.846683 26453 ts_tablet_manager.cc:1918] T ed45ce5bd6424a5aa2402d83be3c1c97 P d9f897aa131b4d91a4cde42bef14dc48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
Aug 08 04:07:13 dist-test-slave-1l3l krb5kdc[25636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.1@KRBTEST.COM
I20260808 04:07:13.847311 26452 tablet_replica.cc:330] T ed45ce5bd6424a5aa2402d83be3c1c97 P 87ca4ca943204f63864c0587ad68d5e4: stopping tablet replica
I20260808 04:07:13.847751 26452 raft_consensus.cc:2229] T ed45ce5bd6424a5aa2402d83be3c1c97 P 87ca4ca943204f63864c0587ad68d5e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:07:13.848327 26452 raft_consensus.cc:2258] T ed45ce5bd6424a5aa2402d83be3c1c97 P 87ca4ca943204f63864c0587ad68d5e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:07:13.848575 26453 log.cc:1196] T ed45ce5bd6424a5aa2402d83be3c1c97 P d9f897aa131b4d91a4cde42bef14dc48: Deleting WAL directory at /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-2/wal/wals/ed45ce5bd6424a5aa2402d83be3c1c97
I20260808 04:07:13.849159 26453 ts_tablet_manager.cc:1939] T ed45ce5bd6424a5aa2402d83be3c1c97 P d9f897aa131b4d91a4cde42bef14dc48: Deleting consensus metadata
I20260808 04:07:13.850464 26452 ts_tablet_manager.cc:1905] T ed45ce5bd6424a5aa2402d83be3c1c97 P 87ca4ca943204f63864c0587ad68d5e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:07:13.851222 25684 catalog_manager.cc:4841] TS d9f897aa131b4d91a4cde42bef14dc48 (127.5.83.3:36921): tablet ed45ce5bd6424a5aa2402d83be3c1c97 (table test-table [id=1bb1f78c0c2d479db4288a884812d1d9]) successfully deleted
I20260808 04:07:13.856104 26410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:13.823562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39736 (local address 127.5.83.1:36007)
0808 04:07:13.823768 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0808 04:07:13.823774 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:07:13.836288 (+ 12514us) server_negotiation.cc:407] Connection header received
0808 04:07:13.836478 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:07:13.836484 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:07:13.836598 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:07:13.836697 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:07:13.838842 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:13.840085 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:13.842105 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:13.842771 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:13.842863 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:13.848436 (+ 5573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:07:13.848451 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:07:13.848470 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:07:13.848515 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:07:13.852824 (+ 4309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:13.853671 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:13.853680 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:13.853684 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:13.853836 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:13.854274 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:13.854279 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:13.854283 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:13.854807 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:07:13.855133 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0808 04:07:13.855449 (+ 316us) server_negotiation.cc:299] Negotiation successful
0808 04:07:13.855683 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 04:07:13.856732 26452 ts_tablet_manager.cc:1918] T ed45ce5bd6424a5aa2402d83be3c1c97 P 87ca4ca943204f63864c0587ad68d5e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:07:13.856873 25980 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:07:13.856901 26452 log.cc:1196] T ed45ce5bd6424a5aa2402d83be3c1c97 P 87ca4ca943204f63864c0587ad68d5e4: Deleting WAL directory at /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-0/wal/wals/ed45ce5bd6424a5aa2402d83be3c1c97
I20260808 04:07:13.945153 26452 ts_tablet_manager.cc:1939] T ed45ce5bd6424a5aa2402d83be3c1c97 P 87ca4ca943204f63864c0587ad68d5e4: Deleting consensus metadata
I20260808 04:07:13.945194 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 25806
I20260808 04:07:13.970041 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 25997
W20260808 04:07:13.970052 25683 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.83.1:36007 (error 108)
I20260808 04:07:13.972074 26407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:13.971559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:36007 (local address 127.0.0.1:39750)
0808 04:07:13.971777 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:13.971897 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.1:36007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260808 04:07:13.972340 25683 catalog_manager.cc:4584] TS 87ca4ca943204f63864c0587ad68d5e4 (127.5.83.1:36007): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ed45ce5bd6424a5aa2402d83be3c1c97: Network error: Client connection negotiation failed: client connection to 127.5.83.1:36007: connect: Connection refused (error 111)
Aug 08 04:07:13 dist-test-slave-1l3l krb5kdc[25636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.2@KRBTEST.COM
I20260808 04:07:13.986562 26408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:13.971339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54024 (local address 127.5.83.2:45521)
0808 04:07:13.971535 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0808 04:07:13.971541 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:07:13.971559 (+ 18us) server_negotiation.cc:407] Connection header received
0808 04:07:13.971629 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:07:13.971633 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:07:13.971756 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:07:13.971851 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:07:13.973172 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:13.973994 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:13.975261 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:13.975833 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:13.975945 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:13.980280 (+ 4335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:07:13.980293 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:07:13.980310 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:07:13.980353 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:07:13.983171 (+ 2818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:13.983992 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:13.984000 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:13.984002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:13.984140 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:13.984532 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:13.984537 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:13.984539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:13.985010 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:07:13.985382 (+ 372us) server_negotiation.cc:1035] Waiting for connection context
0808 04:07:13.985953 (+ 571us) server_negotiation.cc:299] Negotiation successful
0808 04:07:13.986174 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260808 04:07:13.987347 26185 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:07:14.025686 26407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:14.025265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:36007 (local address 127.0.0.1:39764)
0808 04:07:14.025462 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:14.025553 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.1:36007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260808 04:07:14.026679 26407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:14.026371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:36007 (local address 127.0.0.1:39778)
0808 04:07:14.026498 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:14.026563 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.1:36007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260808 04:07:14.087383 26407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:14.086900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:36007 (local address 127.0.0.1:39794)
0808 04:07:14.087117 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:14.087214 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.1:36007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260808 04:07:14.088389 26407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:14.088084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:36007 (local address 127.0.0.1:39800)
0808 04:07:14.088201 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:14.088282 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.1:36007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260808 04:07:14.100472 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 25997
I20260808 04:07:14.126396 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 26202
Aug 08 04:07:14 dist-test-slave-1l3l krb5kdc[25636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.3@KRBTEST.COM
I20260808 04:07:14.141747 26412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:14.127161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51888 (local address 127.5.83.3:36921)
0808 04:07:14.127341 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0808 04:07:14.127347 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:07:14.127450 (+ 103us) server_negotiation.cc:407] Connection header received
0808 04:07:14.127634 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:07:14.127640 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:07:14.127856 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:07:14.127951 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:07:14.129257 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:14.130030 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:14.131454 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:14.131923 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:14.132035 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:14.136525 (+ 4490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:07:14.136541 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:07:14.136559 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:07:14.136593 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:07:14.139102 (+ 2509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:14.139747 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:14.139757 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:14.139761 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:14.139910 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:14.140263 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:14.140268 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:14.140272 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:14.140624 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:07:14.140901 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0808 04:07:14.141219 (+ 318us) server_negotiation.cc:299] Negotiation successful
0808 04:07:14.141434 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 04:07:14.142426 26378 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:07:14.169224 26407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:14.168782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:36007 (local address 127.0.0.1:39806)
0808 04:07:14.168974 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:14.169068 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.1:36007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260808 04:07:14.170251 26407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:14.169849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:36007 (local address 127.0.0.1:39814)
0808 04:07:14.170055 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:14.170138 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.1:36007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260808 04:07:14.230274 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 26202
I20260808 04:07:14.251636 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 25656
I20260808 04:07:14.252825 25754 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:07:14.360554 26407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:14.359938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:36007 (local address 127.0.0.1:39830)
0808 04:07:14.360205 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:14.360318 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.1:36007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153}
I20260808 04:07:14.360539 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 25656
I20260808 04:07:14.387132 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:35721
--webserver_interface=127.5.83.62
--webserver_port=34815
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab
--principal=kudu/127.5.83.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:39647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.62:35721
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:07:14.704846 26465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:07:14.705117 26465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:07:14.705255 26465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:07:14.712167 26465 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:07:14.712267 26465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:07:14.712340 26465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:07:14.712417 26465 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:07:14.712477 26465 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:07:14.721814 26465 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:39647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.5.83.62:35721
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab
--principal=kudu/127.5.83.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:35721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.62
--webserver_port=34815
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:07:14.722975 26465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:07:14.724252 26465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:07:14.746177 26474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:07:14.746204 26476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:07:14.746245 26478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:07:14.746935 26465 server_base.cc:1034] running on GCE node
I20260808 04:07:14.747748 26465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:07:14.749243 26465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:07:14.750463 26465 hybrid_clock.cc:648] HybridClock initialized: now 1786162034750418 us; error 38 us; skew 500 ppm
I20260808 04:07:14.750844 26465 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:07:14 dist-test-slave-1l3l krb5kdc[25636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162034, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:07:14.754472 26465 init.cc:377] Logged in from keytab as kudu/127.5.83.62@KRBTEST.COM (short username kudu)
I20260808 04:07:14.756058 26465 webserver.cc:460] Webserver started at http://127.5.83.62:34815/ using document root <none> and password file <none>
I20260808 04:07:14.756750 26465 fs_manager.cc:359] Metadata directory not provided
I20260808 04:07:14.756886 26465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:07:14.762387 26465 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 04:07:14.765165 26489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:07:14.766502 26465 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 04:07:14.766669 26465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/master-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/master-0/wal
uuid: "231726dbe3ec4fd49e58e8c2a800ed19"
format_stamp: "Formatted at 2026-08-08 04:07:11 on dist-test-slave-1l3l"
server_key: "1c35a255f052a90bdb28ea785819b122"
server_key_iv: "798b3c5a5fe4e2a133f9d9ba66eb5ecd"
server_key_version: "encryptionkey@0"
I20260808 04:07:14.767699 26465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:07:14.801208 26465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:07:14.805383 26465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:07:14.805660 26465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:07:14.823833 26465 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.62:35721
I20260808 04:07:14.823870 26565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.62:35721 every 8 connection(s)
I20260808 04:07:14.825337 26465 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
I20260808 04:07:14.830579 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 26465
I20260808 04:07:14.831583 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.1:36007
--local_ip_for_outbound_sockets=127.5.83.1
--tserver_master_addrs=127.5.83.62:35721
--webserver_port=37729
--webserver_interface=127.5.83.1
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.1.keytab
--principal=kudu/127.5.83.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:39647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:07:14.833184 26566 sys_catalog.cc:263] Verifying existing consensus state
I20260808 04:07:14.837638 26566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 231726dbe3ec4fd49e58e8c2a800ed19: Bootstrap starting.
I20260808 04:07:14.857681 26566 log.cc:824] T 00000000000000000000000000000000 P 231726dbe3ec4fd49e58e8c2a800ed19: Log is configured to *not* fsync() on all Append() calls
I20260808 04:07:14.870003 26566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 231726dbe3ec4fd49e58e8c2a800ed19: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 04:07:14.870519 26566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 231726dbe3ec4fd49e58e8c2a800ed19: Bootstrap complete.
I20260808 04:07:14.879540 26566 raft_consensus.cc:348] T 00000000000000000000000000000000 P 231726dbe3ec4fd49e58e8c2a800ed19 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "231726dbe3ec4fd49e58e8c2a800ed19" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 35721 } }
I20260808 04:07:14.880378 26566 raft_consensus.cc:729] T 00000000000000000000000000000000 P 231726dbe3ec4fd49e58e8c2a800ed19 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 231726dbe3ec4fd49e58e8c2a800ed19, State: Initialized, Role: FOLLOWER
I20260808 04:07:14.880963 26566 consensus_queue.cc:260] T 00000000000000000000000000000000 P 231726dbe3ec4fd49e58e8c2a800ed19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "231726dbe3ec4fd49e58e8c2a800ed19" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 35721 } }
I20260808 04:07:14.881175 26566 raft_consensus.cc:388] T 00000000000000000000000000000000 P 231726dbe3ec4fd49e58e8c2a800ed19 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:07:14.881289 26566 raft_consensus.cc:482] T 00000000000000000000000000000000 P 231726dbe3ec4fd49e58e8c2a800ed19 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:07:14.881430 26566 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 231726dbe3ec4fd49e58e8c2a800ed19 [term 1 FOLLOWER]: Advancing to term 2
I20260808 04:07:14.885178 26566 raft_consensus.cc:504] T 00000000000000000000000000000000 P 231726dbe3ec4fd49e58e8c2a800ed19 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "231726dbe3ec4fd49e58e8c2a800ed19" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 35721 } }
I20260808 04:07:14.885649 26566 leader_election.cc:302] T 00000000000000000000000000000000 P 231726dbe3ec4fd49e58e8c2a800ed19 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 231726dbe3ec4fd49e58e8c2a800ed19; no voters:
I20260808 04:07:14.886260 26566 leader_election.cc:288] T 00000000000000000000000000000000 P 231726dbe3ec4fd49e58e8c2a800ed19 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 04:07:14.886404 26572 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 231726dbe3ec4fd49e58e8c2a800ed19 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 04:07:14.887560 26572 raft_consensus.cc:686] T 00000000000000000000000000000000 P 231726dbe3ec4fd49e58e8c2a800ed19 [term 2 LEADER]: Becoming Leader. State: Replica: 231726dbe3ec4fd49e58e8c2a800ed19, State: Running, Role: LEADER
I20260808 04:07:14.888163 26572 consensus_queue.cc:237] T 00000000000000000000000000000000 P 231726dbe3ec4fd49e58e8c2a800ed19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "231726dbe3ec4fd49e58e8c2a800ed19" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 35721 } }
I20260808 04:07:14.888509 26566 sys_catalog.cc:564] T 00000000000000000000000000000000 P 231726dbe3ec4fd49e58e8c2a800ed19 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:07:14.890571 26575 sys_catalog.cc:455] T 00000000000000000000000000000000 P 231726dbe3ec4fd49e58e8c2a800ed19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 231726dbe3ec4fd49e58e8c2a800ed19. Latest consensus state: current_term: 2 leader_uuid: "231726dbe3ec4fd49e58e8c2a800ed19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "231726dbe3ec4fd49e58e8c2a800ed19" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 35721 } } }
I20260808 04:07:14.890954 26575 sys_catalog.cc:458] T 00000000000000000000000000000000 P 231726dbe3ec4fd49e58e8c2a800ed19 [sys.catalog]: This master's current role is: LEADER
I20260808 04:07:14.891534 26574 sys_catalog.cc:455] T 00000000000000000000000000000000 P 231726dbe3ec4fd49e58e8c2a800ed19 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "231726dbe3ec4fd49e58e8c2a800ed19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "231726dbe3ec4fd49e58e8c2a800ed19" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 35721 } } }
I20260808 04:07:14.891830 26574 sys_catalog.cc:458] T 00000000000000000000000000000000 P 231726dbe3ec4fd49e58e8c2a800ed19 [sys.catalog]: This master's current role is: LEADER
I20260808 04:07:14.901181 26579 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:07:14.909385 26579 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:07:14.910801 26579 catalog_manager.cc:1259] Loaded cluster ID: f2cf49dc4abb44b39ec9cf72bb35b00e
I20260808 04:07:14.910943 26579 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:07:14.914624 26579 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260808 04:07:14.932257 26579 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260808 04:07:14.954397 26595 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260808 04:07:14.956401 26579 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:07:14.958057 26579 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 231726dbe3ec4fd49e58e8c2a800ed19: Loaded TSK: 0
I20260808 04:07:14.959468 26579 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 04:07:15.178917 26568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:07:15.179183 26568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:07:15.179320 26568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:07:15.186817 26568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:07:15.186950 26568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:07:15.187114 26568 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.1
I20260808 04:07:15.196674 26568 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:39647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.1.keytab
--principal=kudu/127.5.83.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.1:36007
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.83.1
--webserver_port=37729
--tserver_master_addrs=127.5.83.62:35721
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.1
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:07:15.197881 26568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:07:15.199126 26568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:07:15.213138 26604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:07:15.213315 26603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:07:15.214475 26568 server_base.cc:1034] running on GCE node
W20260808 04:07:15.215927 26606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:07:15.216563 26568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:07:15.217474 26568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:07:15.218683 26568 hybrid_clock.cc:648] HybridClock initialized: now 1786162035218643 us; error 38 us; skew 500 ppm
I20260808 04:07:15.218961 26568 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:07:15 dist-test-slave-1l3l krb5kdc[25636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162035, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:07:15.222157 26568 init.cc:377] Logged in from keytab as kudu/127.5.83.1@KRBTEST.COM (short username kudu)
I20260808 04:07:15.223559 26568 webserver.cc:460] Webserver started at http://127.5.83.1:37729/ using document root <none> and password file <none>
I20260808 04:07:15.224223 26568 fs_manager.cc:359] Metadata directory not provided
I20260808 04:07:15.224355 26568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:07:15.229326 26568 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 04:07:15.231911 26616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:07:15.233186 26568 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:07:15.233325 26568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-0/wal
uuid: "87ca4ca943204f63864c0587ad68d5e4"
format_stamp: "Formatted at 2026-08-08 04:07:12 on dist-test-slave-1l3l"
server_key: "6003d2fd8dc22147facf5cada69e4412"
server_key_iv: "2193ebbc6e935dc13dc2eb0474e0a27f"
server_key_version: "encryptionkey@0"
I20260808 04:07:15.234205 26568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:07:15.254879 26568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:07:15.258828 26568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:07:15.259068 26568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:07:15.260046 26568 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:07:15.261853 26626 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260808 04:07:15.266934 26626 ts_tablet_manager.cc:1852] T ed45ce5bd6424a5aa2402d83be3c1c97 P 87ca4ca943204f63864c0587ad68d5e4: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260808 04:07:15.267913 26626 ts_tablet_manager.cc:1905] T ed45ce5bd6424a5aa2402d83be3c1c97 P 87ca4ca943204f63864c0587ad68d5e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:07:15.274751 26626 ts_tablet_manager.cc:1918] T ed45ce5bd6424a5aa2402d83be3c1c97 P 87ca4ca943204f63864c0587ad68d5e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:07:15.274999 26626 ts_tablet_manager.cc:1939] T ed45ce5bd6424a5aa2402d83be3c1c97 P 87ca4ca943204f63864c0587ad68d5e4: Deleting consensus metadata
I20260808 04:07:15.275447 26568 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260808 04:07:15.275552 26568 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.014s user 0.000s sys 0.001s
I20260808 04:07:15.275657 26568 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:07:15.275746 26568 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:07:15.306732 26568 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.1:36007
I20260808 04:07:15.306795 26797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.1:36007 every 8 connection(s)
I20260808 04:07:15.307957 26568 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
I20260808 04:07:15.312673 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 26568
I20260808 04:07:15.313277 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.2:45521
--local_ip_for_outbound_sockets=127.5.83.2
--tserver_master_addrs=127.5.83.62:35721
--webserver_port=41049
--webserver_interface=127.5.83.2
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.2.keytab
--principal=kudu/127.5.83.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:39647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 04:07:15 dist-test-slave-1l3l krb5kdc[25636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162035, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.1@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:07:15.331087 26801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:15.310949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:35721 (local address 127.5.83.1:53397)
0808 04:07:15.311946 (+ 997us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:15.311991 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 04:07:15.313811 (+ 1820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:07:15.315285 (+ 1474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:07:15.315293 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:07:15.315882 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:07:15.316418 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:07:15.316437 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:07:15.318537 (+ 2100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:07:15.318541 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:07:15.319816 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:07:15.319829 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:07:15.321022 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:07:15.321026 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:07:15.321202 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:15.321806 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:07:15.321827 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:07:15.323934 (+ 2107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:07:15.327108 (+ 3174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:07:15.327116 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:07:15.327129 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:07:15.327545 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:07:15.327993 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:07:15.327998 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:07:15.328002 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:07:15.328179 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:07:15.328851 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:07:15.328859 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:07:15.329155 (+ 296us) client_negotiation.cc:769] Sending connection context
0808 04:07:15.329379 (+ 224us) client_negotiation.cc:240] Negotiation successful
0808 04:07:15.329856 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":687,"thread_start_us":472,"threads_started":1}
I20260808 04:07:15.331097 26800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:15.310680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.1:53397 (local address 127.5.83.62:35721)
0808 04:07:15.311419 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0808 04:07:15.311434 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 04:07:15.312120 (+ 686us) server_negotiation.cc:407] Connection header received
0808 04:07:15.314363 (+ 2243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:07:15.314381 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:07:15.314819 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:07:15.315186 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:07:15.316641 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:15.318374 (+ 1733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:15.319960 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:15.320841 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:15.321025 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:15.324175 (+ 3150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:07:15.324182 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:07:15.324194 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:07:15.324228 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:07:15.326979 (+ 2751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:15.327680 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:15.327686 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:15.327691 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:15.327883 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:15.328295 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:15.328299 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:15.328301 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:15.328713 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:07:15.328911 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0808 04:07:15.329600 (+ 689us) server_negotiation.cc:299] Negotiation successful
0808 04:07:15.329993 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":316,"threads_started":1}
I20260808 04:07:15.334126 26798 heartbeater.cc:347] Connected to a master server at 127.5.83.62:35721
I20260808 04:07:15.334443 26798 heartbeater.cc:464] Registering TS with master...
I20260808 04:07:15.335281 26798 heartbeater.cc:511] Master 127.5.83.62:35721 requested a full tablet report, sending...
I20260808 04:07:15.337532 26515 ts_manager.cc:194] Registered new tserver with Master: 87ca4ca943204f63864c0587ad68d5e4 (127.5.83.1:36007)
I20260808 04:07:15.339063 26515 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.1@KRBTEST.COM'} at 127.5.83.1:53397
W20260808 04:07:15.640239 26803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:07:15.640528 26803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:07:15.640799 26803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:07:15.648542 26803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:07:15.648644 26803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:07:15.648797 26803 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.2
I20260808 04:07:15.658555 26803 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:39647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.2.keytab
--principal=kudu/127.5.83.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.2:45521
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.83.2
--webserver_port=41049
--tserver_master_addrs=127.5.83.62:35721
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.2
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:07:15.659822 26803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:07:15.661125 26803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:07:15.673547 26808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:07:15.673488 26811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:07:15.673516 26809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:07:15.674628 26803 server_base.cc:1034] running on GCE node
I20260808 04:07:15.675192 26803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:07:15.676165 26803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:07:15.677412 26803 hybrid_clock.cc:648] HybridClock initialized: now 1786162035677358 us; error 43 us; skew 500 ppm
I20260808 04:07:15.677704 26803 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:07:15 dist-test-slave-1l3l krb5kdc[25636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162035, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:07:15.680706 26803 init.cc:377] Logged in from keytab as kudu/127.5.83.2@KRBTEST.COM (short username kudu)
I20260808 04:07:15.681994 26803 webserver.cc:460] Webserver started at http://127.5.83.2:41049/ using document root <none> and password file <none>
I20260808 04:07:15.682634 26803 fs_manager.cc:359] Metadata directory not provided
I20260808 04:07:15.682761 26803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:07:15.687795 26803 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260808 04:07:15.690371 26830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:07:15.691581 26803 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 04:07:15.691749 26803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-1/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-1/wal
uuid: "37ad64cb60b94c699ebb75ac30c3865c"
format_stamp: "Formatted at 2026-08-08 04:07:13 on dist-test-slave-1l3l"
server_key: "04f99e3d10123a3a0e3abe6a15c18711"
server_key_iv: "423165bcd7fceddee3de5393e2d68096"
server_key_version: "encryptionkey@0"
I20260808 04:07:15.692682 26803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:07:15.721127 26803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:07:15.725219 26803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:07:15.725486 26803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:07:15.726406 26803 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:07:15.727679 26803 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:07:15.727798 26803 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:07:15.727903 26803 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:07:15.727980 26803 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:07:15.758324 26803 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.2:45521
I20260808 04:07:15.758375 27002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.2:45521 every 8 connection(s)
I20260808 04:07:15.759661 26803 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
I20260808 04:07:15.769342 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 26803
I20260808 04:07:15.769927 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.3:36921
--local_ip_for_outbound_sockets=127.5.83.3
--tserver_master_addrs=127.5.83.62:35721
--webserver_port=46075
--webserver_interface=127.5.83.3
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.3.keytab
--principal=kudu/127.5.83.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:39647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 04:07:15 dist-test-slave-1l3l krb5kdc[25636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162035, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.2@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:07:15.778249 26800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:15.762089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.2:56579 (local address 127.5.83.62:35721)
0808 04:07:15.762255 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0808 04:07:15.762259 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:07:15.763351 (+ 1092us) server_negotiation.cc:407] Connection header received
0808 04:07:15.765147 (+ 1796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:07:15.765153 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:07:15.765232 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:07:15.765319 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:07:15.766715 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:15.767416 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:15.768982 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:15.769414 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:15.769527 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:15.772940 (+ 3413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:07:15.772945 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:07:15.772949 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:07:15.772982 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:07:15.775233 (+ 2251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:15.776057 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:15.776062 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:15.776064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:15.776197 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:15.776613 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:15.776616 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:15.776618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:15.776890 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:07:15.777023 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0808 04:07:15.777683 (+ 660us) server_negotiation.cc:299] Negotiation successful
0808 04:07:15.777926 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 04:07:15.779126 27006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:15.762537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:35721 (local address 127.5.83.2:56579)
0808 04:07:15.763197 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:15.763238 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 04:07:15.764892 (+ 1654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:07:15.765555 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:07:15.765563 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:07:15.766026 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:07:15.766550 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:07:15.766567 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:07:15.767573 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:07:15.767577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:07:15.768818 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:07:15.768831 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:07:15.769660 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:07:15.769668 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:07:15.769861 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:15.770505 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:07:15.770526 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:07:15.772740 (+ 2214us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:07:15.775368 (+ 2628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:07:15.775377 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:07:15.775399 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:07:15.775936 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:07:15.776305 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:07:15.776310 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:07:15.776314 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:07:15.776498 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:07:15.777013 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:07:15.777020 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:07:15.777311 (+ 291us) client_negotiation.cc:769] Sending connection context
0808 04:07:15.777549 (+ 238us) client_negotiation.cc:240] Negotiation successful
0808 04:07:15.778021 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":271,"threads_started":1}
I20260808 04:07:15.780512 27004 heartbeater.cc:347] Connected to a master server at 127.5.83.62:35721
I20260808 04:07:15.780800 27004 heartbeater.cc:464] Registering TS with master...
I20260808 04:07:15.781461 27004 heartbeater.cc:511] Master 127.5.83.62:35721 requested a full tablet report, sending...
I20260808 04:07:15.782928 26515 ts_manager.cc:194] Registered new tserver with Master: 37ad64cb60b94c699ebb75ac30c3865c (127.5.83.2:45521)
I20260808 04:07:15.784009 26515 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.2@KRBTEST.COM'} at 127.5.83.2:56579
W20260808 04:07:16.093281 27007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:07:16.093533 27007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:07:16.093626 27007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:07:16.100843 27007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:07:16.100947 27007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:07:16.101064 27007 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.3
I20260808 04:07:16.110421 27007 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:39647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.3.keytab
--principal=kudu/127.5.83.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.3:36921
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.83.3
--webserver_port=46075
--tserver_master_addrs=127.5.83.62:35721
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.3
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:07:16.111538 27007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:07:16.112792 27007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:07:16.125324 27017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:07:16.125368 27020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:07:16.125386 27016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:07:16.126371 27007 server_base.cc:1034] running on GCE node
I20260808 04:07:16.126883 27007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:07:16.127661 27007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:07:16.128890 27007 hybrid_clock.cc:648] HybridClock initialized: now 1786162036128830 us; error 46 us; skew 500 ppm
I20260808 04:07:16.129176 27007 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:07:16 dist-test-slave-1l3l krb5kdc[25636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162036, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:07:16.132196 27007 init.cc:377] Logged in from keytab as kudu/127.5.83.3@KRBTEST.COM (short username kudu)
I20260808 04:07:16.133476 27007 webserver.cc:460] Webserver started at http://127.5.83.3:46075/ using document root <none> and password file <none>
I20260808 04:07:16.134075 27007 fs_manager.cc:359] Metadata directory not provided
I20260808 04:07:16.134186 27007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:07:16.139216 27007 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.002s
I20260808 04:07:16.141701 27030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:07:16.142935 27007 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 04:07:16.143064 27007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-2/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-2/wal
uuid: "d9f897aa131b4d91a4cde42bef14dc48"
format_stamp: "Formatted at 2026-08-08 04:07:13 on dist-test-slave-1l3l"
server_key: "917e5a9e738808ecfa06cc7e9cbe0ddc"
server_key_iv: "ba4ec26d2ac5cec225a0c6ec200e4d02"
server_key_version: "encryptionkey@0"
I20260808 04:07:16.144171 27007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:07:16.168843 27007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:07:16.172688 27007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:07:16.172930 27007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:07:16.173868 27007 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:07:16.175109 27007 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:07:16.175199 27007 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:07:16.175316 27007 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:07:16.175379 27007 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:07:16.202922 27007 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.3:36921
I20260808 04:07:16.203033 27198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.3:36921 every 8 connection(s)
I20260808 04:07:16.204259 27007 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
I20260808 04:07:16.206691 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 27007
Aug 08 04:07:16 dist-test-slave-1l3l krb5kdc[25636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162036, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.3@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:07:16.226322 27206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:16.207622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33358 (local address 127.5.83.62:35721)
0808 04:07:16.212790 (+ 5168us) server_negotiation.cc:206] Beginning negotiation
0808 04:07:16.212794 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:07:16.212807 (+ 13us) server_negotiation.cc:407] Connection header received
0808 04:07:16.212849 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:07:16.212852 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:07:16.212911 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:07:16.212989 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:07:16.214677 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:16.215517 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:16.217386 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:16.217787 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:16.217902 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:16.220636 (+ 2734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:07:16.220641 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:07:16.220644 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:07:16.220689 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:07:16.223560 (+ 2871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:16.224298 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:16.224302 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:16.224305 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:16.224498 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:16.224923 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:16.224926 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:16.224929 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:16.225194 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:07:16.225481 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0808 04:07:16.225724 (+ 243us) server_negotiation.cc:299] Negotiation successful
0808 04:07:16.225941 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5080,"thread_start_us":4965,"threads_started":1}
I20260808 04:07:16.226694 26800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:16.206669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.3:51341 (local address 127.5.83.62:35721)
0808 04:07:16.206882 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0808 04:07:16.206889 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:07:16.208579 (+ 1690us) server_negotiation.cc:407] Connection header received
0808 04:07:16.211759 (+ 3180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:07:16.211766 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:07:16.211872 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:07:16.211981 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:07:16.213695 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:16.214429 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:16.216493 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:16.216874 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:16.216972 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:16.220553 (+ 3581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:07:16.220560 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:07:16.220565 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:07:16.220601 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:07:16.223481 (+ 2880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:16.224226 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:16.224232 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:16.224235 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:16.224352 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:16.224826 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:16.224830 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:16.224834 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:16.225139 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:07:16.225246 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0808 04:07:16.226123 (+ 877us) server_negotiation.cc:299] Negotiation successful
0808 04:07:16.226317 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260808 04:07:16.227967 27203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:16.207090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:35721 (local address 127.5.83.3:51341)
0808 04:07:16.208398 (+ 1308us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:16.208464 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0808 04:07:16.211387 (+ 2923us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:07:16.212333 (+ 946us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:07:16.212342 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:07:16.212806 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:07:16.213511 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:07:16.213537 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:07:16.214591 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:07:16.214596 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:07:16.216363 (+ 1767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:07:16.216379 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:07:16.217093 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:07:16.217101 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:07:16.217369 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:16.217989 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:07:16.218012 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:07:16.220346 (+ 2334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:07:16.223620 (+ 3274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:07:16.223628 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:07:16.223644 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:07:16.224102 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:07:16.224479 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:07:16.224483 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:07:16.224487 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:07:16.224710 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:07:16.225281 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:07:16.225289 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:07:16.225668 (+ 379us) client_negotiation.cc:769] Sending connection context
0808 04:07:16.226386 (+ 718us) client_negotiation.cc:240] Negotiation successful
0808 04:07:16.226801 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":947,"thread_start_us":744,"threads_started":1}
I20260808 04:07:16.229215 27199 heartbeater.cc:347] Connected to a master server at 127.5.83.62:35721
I20260808 04:07:16.229470 27199 heartbeater.cc:464] Registering TS with master...
I20260808 04:07:16.230083 27199 heartbeater.cc:511] Master 127.5.83.62:35721 requested a full tablet report, sending...
I20260808 04:07:16.231356 26514 ts_manager.cc:194] Registered new tserver with Master: d9f897aa131b4d91a4cde42bef14dc48 (127.5.83.3:36921)
I20260808 04:07:16.232290 26514 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.3@KRBTEST.COM'} at 127.5.83.3:51341
I20260808 04:07:16.240887 5452 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:07:16.246941 26800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:16.241641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33370 (local address 127.5.83.62:35721)
0808 04:07:16.241826 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0808 04:07:16.241832 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:07:16.242221 (+ 389us) server_negotiation.cc:407] Connection header received
0808 04:07:16.242369 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:07:16.242373 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:07:16.242500 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:07:16.242610 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:07:16.242953 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:16.243657 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:16.245067 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:16.245474 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:16.245559 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:16.246035 (+ 476us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:07:16.246455 (+ 420us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:07:16.246551 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0808 04:07:16.246673 (+ 122us) server_negotiation.cc:299] Negotiation successful
0808 04:07:16.246729 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 04:07:16.247813 26514 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:33370:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:07:16.254779 26514 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:07:16.277344 27214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:16.266589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54034 (local address 127.5.83.2:45521)
0808 04:07:16.267222 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0808 04:07:16.267231 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 04:07:16.268339 (+ 1108us) server_negotiation.cc:407] Connection header received
0808 04:07:16.268671 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:07:16.268680 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:07:16.268923 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:07:16.269085 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:07:16.270392 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:16.271651 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:16.274738 (+ 3087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:16.276175 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:16.276298 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:16.276527 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0808 04:07:16.276872 (+ 345us) server_negotiation.cc:299] Negotiation successful
0808 04:07:16.277001 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":628,"threads_started":1}
I20260808 04:07:16.277369 27220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:16.266149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.2:45521 (local address 127.0.0.1:54034)
0808 04:07:16.268172 (+ 2023us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:16.268250 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0808 04:07:16.268437 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:07:16.269741 (+ 1304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:07:16.269747 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:07:16.269774 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:07:16.270186 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:07:16.270198 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:07:16.271919 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:07:16.271924 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:07:16.274582 (+ 2658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:07:16.274597 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:07:16.276375 (+ 1778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:07:16.276382 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:07:16.276572 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:16.276626 (+ 54us) client_negotiation.cc:769] Sending connection context
0808 04:07:16.276763 (+ 137us) client_negotiation.cc:240] Negotiation successful
0808 04:07:16.276951 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1773,"mutex_wait_us":90,"thread_start_us":1630,"threads_started":1}
I20260808 04:07:16.277601 27216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:16.266663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51896 (local address 127.5.83.3:36921)
0808 04:07:16.270835 (+ 4172us) server_negotiation.cc:206] Beginning negotiation
0808 04:07:16.270843 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:07:16.270872 (+ 29us) server_negotiation.cc:407] Connection header received
0808 04:07:16.270957 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:07:16.270964 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:07:16.271158 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:07:16.271323 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:07:16.272046 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:16.273276 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:16.275162 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:16.276515 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:16.276647 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:16.276851 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0808 04:07:16.277127 (+ 276us) server_negotiation.cc:299] Negotiation successful
0808 04:07:16.277276 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4004,"thread_start_us":3848,"threads_started":1}
I20260808 04:07:16.278051 27218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:16.266823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39838 (local address 127.5.83.1:36007)
0808 04:07:16.267618 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0808 04:07:16.267627 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 04:07:16.269091 (+ 1464us) server_negotiation.cc:407] Connection header received
0808 04:07:16.269369 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:07:16.269376 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:07:16.269580 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:07:16.269729 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:07:16.270543 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:16.272027 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:16.274785 (+ 2758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:16.276075 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:16.276197 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:16.276402 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0808 04:07:16.277633 (+ 1231us) server_negotiation.cc:299] Negotiation successful
0808 04:07:16.277793 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":561,"threads_started":1}
I20260808 04:07:16.278461 27219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:16.266175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.3:36921 (local address 127.0.0.1:51896)
0808 04:07:16.268172 (+ 1997us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:16.268244 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0808 04:07:16.268401 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:07:16.271338 (+ 2937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:07:16.271343 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:07:16.271368 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:07:16.271774 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:07:16.271836 (+ 62us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:07:16.273408 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:07:16.273411 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:07:16.274993 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:07:16.275007 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:07:16.276729 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:07:16.276735 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:07:16.276952 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:16.276981 (+ 29us) client_negotiation.cc:769] Sending connection context
0808 04:07:16.277075 (+ 94us) client_negotiation.cc:240] Negotiation successful
0808 04:07:16.277189 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1806,"mutex_wait_us":42,"thread_start_us":2212,"threads_started":1}
I20260808 04:07:16.278781 27221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:16.266953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:36007 (local address 127.0.0.1:39838)
0808 04:07:16.268973 (+ 2020us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:16.268994 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 04:07:16.269189 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:07:16.269971 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:07:16.269976 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:07:16.269998 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:07:16.270366 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:07:16.270377 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:07:16.272195 (+ 1818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:07:16.272200 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:07:16.274590 (+ 2390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:07:16.274600 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:07:16.276573 (+ 1973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:07:16.276580 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:07:16.277428 (+ 848us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:16.277460 (+ 32us) client_negotiation.cc:769] Sending connection context
0808 04:07:16.278255 (+ 795us) client_negotiation.cc:240] Negotiation successful
0808 04:07:16.278386 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1892,"mutex_wait_us":759,"thread_start_us":1722,"threads_started":1}
I20260808 04:07:16.280627 26695 tablet_service.cc:1467] Processing CreateTablet for tablet 3cd5eda73f1c4ef29f04019a07a67e7b (DEFAULT_TABLE table=test-table [id=4e89e46f1c02454d9b47d07a0df70816]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:07:16.281319 26907 tablet_service.cc:1467] Processing CreateTablet for tablet 3cd5eda73f1c4ef29f04019a07a67e7b (DEFAULT_TABLE table=test-table [id=4e89e46f1c02454d9b47d07a0df70816]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:07:16.282083 26695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cd5eda73f1c4ef29f04019a07a67e7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:07:16.282397 26907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cd5eda73f1c4ef29f04019a07a67e7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:07:16.282502 27100 tablet_service.cc:1467] Processing CreateTablet for tablet 3cd5eda73f1c4ef29f04019a07a67e7b (DEFAULT_TABLE table=test-table [id=4e89e46f1c02454d9b47d07a0df70816]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:07:16.283798 27100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cd5eda73f1c4ef29f04019a07a67e7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:07:16.293820 27223 tablet_bootstrap.cc:492] T 3cd5eda73f1c4ef29f04019a07a67e7b P 87ca4ca943204f63864c0587ad68d5e4: Bootstrap starting.
I20260808 04:07:16.297498 27225 tablet_bootstrap.cc:492] T 3cd5eda73f1c4ef29f04019a07a67e7b P d9f897aa131b4d91a4cde42bef14dc48: Bootstrap starting.
I20260808 04:07:16.297853 27223 tablet_bootstrap.cc:654] T 3cd5eda73f1c4ef29f04019a07a67e7b P 87ca4ca943204f63864c0587ad68d5e4: Neither blocks nor log segments found. Creating new log.
I20260808 04:07:16.299203 27223 log.cc:824] T 3cd5eda73f1c4ef29f04019a07a67e7b P 87ca4ca943204f63864c0587ad68d5e4: Log is configured to *not* fsync() on all Append() calls
I20260808 04:07:16.300559 27224 tablet_bootstrap.cc:492] T 3cd5eda73f1c4ef29f04019a07a67e7b P 37ad64cb60b94c699ebb75ac30c3865c: Bootstrap starting.
I20260808 04:07:16.300982 27225 tablet_bootstrap.cc:654] T 3cd5eda73f1c4ef29f04019a07a67e7b P d9f897aa131b4d91a4cde42bef14dc48: Neither blocks nor log segments found. Creating new log.
I20260808 04:07:16.301975 27225 log.cc:824] T 3cd5eda73f1c4ef29f04019a07a67e7b P d9f897aa131b4d91a4cde42bef14dc48: Log is configured to *not* fsync() on all Append() calls
I20260808 04:07:16.304749 27225 tablet_bootstrap.cc:492] T 3cd5eda73f1c4ef29f04019a07a67e7b P d9f897aa131b4d91a4cde42bef14dc48: No bootstrap required, opened a new log
I20260808 04:07:16.304991 27225 ts_tablet_manager.cc:1397] T 3cd5eda73f1c4ef29f04019a07a67e7b P d9f897aa131b4d91a4cde42bef14dc48: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260808 04:07:16.305217 27224 tablet_bootstrap.cc:654] T 3cd5eda73f1c4ef29f04019a07a67e7b P 37ad64cb60b94c699ebb75ac30c3865c: Neither blocks nor log segments found. Creating new log.
I20260808 04:07:16.307358 27224 log.cc:824] T 3cd5eda73f1c4ef29f04019a07a67e7b P 37ad64cb60b94c699ebb75ac30c3865c: Log is configured to *not* fsync() on all Append() calls
I20260808 04:07:16.308853 27223 tablet_bootstrap.cc:492] T 3cd5eda73f1c4ef29f04019a07a67e7b P 87ca4ca943204f63864c0587ad68d5e4: No bootstrap required, opened a new log
I20260808 04:07:16.309119 27223 ts_tablet_manager.cc:1397] T 3cd5eda73f1c4ef29f04019a07a67e7b P 87ca4ca943204f63864c0587ad68d5e4: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.007s
I20260808 04:07:16.310210 27224 tablet_bootstrap.cc:492] T 3cd5eda73f1c4ef29f04019a07a67e7b P 37ad64cb60b94c699ebb75ac30c3865c: No bootstrap required, opened a new log
I20260808 04:07:16.310493 27224 ts_tablet_manager.cc:1397] T 3cd5eda73f1c4ef29f04019a07a67e7b P 37ad64cb60b94c699ebb75ac30c3865c: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.002s
I20260808 04:07:16.316736 27225 raft_consensus.cc:348] T 3cd5eda73f1c4ef29f04019a07a67e7b P d9f897aa131b4d91a4cde42bef14dc48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ad64cb60b94c699ebb75ac30c3865c" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 45521 } } peers { permanent_uuid: "d9f897aa131b4d91a4cde42bef14dc48" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 36921 } } peers { permanent_uuid: "87ca4ca943204f63864c0587ad68d5e4" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 36007 } }
I20260808 04:07:16.317147 27225 raft_consensus.cc:374] T 3cd5eda73f1c4ef29f04019a07a67e7b P d9f897aa131b4d91a4cde42bef14dc48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:07:16.317245 27225 raft_consensus.cc:729] T 3cd5eda73f1c4ef29f04019a07a67e7b P d9f897aa131b4d91a4cde42bef14dc48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9f897aa131b4d91a4cde42bef14dc48, State: Initialized, Role: FOLLOWER
I20260808 04:07:16.317844 27225 consensus_queue.cc:260] T 3cd5eda73f1c4ef29f04019a07a67e7b P d9f897aa131b4d91a4cde42bef14dc48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ad64cb60b94c699ebb75ac30c3865c" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 45521 } } peers { permanent_uuid: "d9f897aa131b4d91a4cde42bef14dc48" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 36921 } } peers { permanent_uuid: "87ca4ca943204f63864c0587ad68d5e4" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 36007 } }
I20260808 04:07:16.319155 27224 raft_consensus.cc:348] T 3cd5eda73f1c4ef29f04019a07a67e7b P 37ad64cb60b94c699ebb75ac30c3865c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ad64cb60b94c699ebb75ac30c3865c" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 45521 } } peers { permanent_uuid: "d9f897aa131b4d91a4cde42bef14dc48" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 36921 } } peers { permanent_uuid: "87ca4ca943204f63864c0587ad68d5e4" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 36007 } }
I20260808 04:07:16.319626 27224 raft_consensus.cc:374] T 3cd5eda73f1c4ef29f04019a07a67e7b P 37ad64cb60b94c699ebb75ac30c3865c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:07:16.319794 27224 raft_consensus.cc:729] T 3cd5eda73f1c4ef29f04019a07a67e7b P 37ad64cb60b94c699ebb75ac30c3865c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37ad64cb60b94c699ebb75ac30c3865c, State: Initialized, Role: FOLLOWER
I20260808 04:07:16.320379 27224 consensus_queue.cc:260] T 3cd5eda73f1c4ef29f04019a07a67e7b P 37ad64cb60b94c699ebb75ac30c3865c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ad64cb60b94c699ebb75ac30c3865c" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 45521 } } peers { permanent_uuid: "d9f897aa131b4d91a4cde42bef14dc48" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 36921 } } peers { permanent_uuid: "87ca4ca943204f63864c0587ad68d5e4" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 36007 } }
I20260808 04:07:16.320565 27223 raft_consensus.cc:348] T 3cd5eda73f1c4ef29f04019a07a67e7b P 87ca4ca943204f63864c0587ad68d5e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ad64cb60b94c699ebb75ac30c3865c" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 45521 } } peers { permanent_uuid: "d9f897aa131b4d91a4cde42bef14dc48" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 36921 } } peers { permanent_uuid: "87ca4ca943204f63864c0587ad68d5e4" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 36007 } }
I20260808 04:07:16.320996 27223 raft_consensus.cc:374] T 3cd5eda73f1c4ef29f04019a07a67e7b P 87ca4ca943204f63864c0587ad68d5e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:07:16.321101 27223 raft_consensus.cc:729] T 3cd5eda73f1c4ef29f04019a07a67e7b P 87ca4ca943204f63864c0587ad68d5e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ca4ca943204f63864c0587ad68d5e4, State: Initialized, Role: FOLLOWER
I20260808 04:07:16.321774 27223 consensus_queue.cc:260] T 3cd5eda73f1c4ef29f04019a07a67e7b P 87ca4ca943204f63864c0587ad68d5e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ad64cb60b94c699ebb75ac30c3865c" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 45521 } } peers { permanent_uuid: "d9f897aa131b4d91a4cde42bef14dc48" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 36921 } } peers { permanent_uuid: "87ca4ca943204f63864c0587ad68d5e4" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 36007 } }
I20260808 04:07:16.325037 27199 heartbeater.cc:503] Master 127.5.83.62:35721 was elected leader, sending a full tablet report...
I20260808 04:07:16.325707 27225 ts_tablet_manager.cc:1428] T 3cd5eda73f1c4ef29f04019a07a67e7b P d9f897aa131b4d91a4cde42bef14dc48: Time spent starting tablet: real 0.021s user 0.013s sys 0.005s
I20260808 04:07:16.326440 26798 heartbeater.cc:503] Master 127.5.83.62:35721 was elected leader, sending a full tablet report...
I20260808 04:07:16.326992 27223 ts_tablet_manager.cc:1428] T 3cd5eda73f1c4ef29f04019a07a67e7b P 87ca4ca943204f63864c0587ad68d5e4: Time spent starting tablet: real 0.018s user 0.013s sys 0.004s
I20260808 04:07:16.328243 27004 heartbeater.cc:503] Master 127.5.83.62:35721 was elected leader, sending a full tablet report...
I20260808 04:07:16.328940 27224 ts_tablet_manager.cc:1428] T 3cd5eda73f1c4ef29f04019a07a67e7b P 37ad64cb60b94c699ebb75ac30c3865c: Time spent starting tablet: real 0.018s user 0.010s sys 0.007s
I20260808 04:07:16.388782 27231 raft_consensus.cc:482] T 3cd5eda73f1c4ef29f04019a07a67e7b P 37ad64cb60b94c699ebb75ac30c3865c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:07:16.389099 27231 raft_consensus.cc:504] T 3cd5eda73f1c4ef29f04019a07a67e7b P 37ad64cb60b94c699ebb75ac30c3865c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ad64cb60b94c699ebb75ac30c3865c" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 45521 } } peers { permanent_uuid: "d9f897aa131b4d91a4cde42bef14dc48" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 36921 } } peers { permanent_uuid: "87ca4ca943204f63864c0587ad68d5e4" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 36007 } }
I20260808 04:07:16.390733 27231 leader_election.cc:288] T 3cd5eda73f1c4ef29f04019a07a67e7b P 37ad64cb60b94c699ebb75ac30c3865c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9f897aa131b4d91a4cde42bef14dc48 (127.5.83.3:36921), 87ca4ca943204f63864c0587ad68d5e4 (127.5.83.1:36007)
I20260808 04:07:16.399017 27216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:16.391121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.2:45023 (local address 127.5.83.3:36921)
0808 04:07:16.391336 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0808 04:07:16.391343 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:07:16.392261 (+ 918us) server_negotiation.cc:407] Connection header received
0808 04:07:16.392428 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:07:16.392433 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:07:16.392502 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:07:16.392569 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:07:16.393141 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:16.394164 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:16.396761 (+ 2597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:16.398019 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:16.398155 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:16.398257 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0808 04:07:16.398636 (+ 379us) server_negotiation.cc:299] Negotiation successful
0808 04:07:16.398764 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 04:07:16.399076 27236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:16.390965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.3:36921 (local address 127.5.83.2:45023)
0808 04:07:16.392086 (+ 1121us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:16.392100 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:07:16.392302 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:07:16.392611 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:07:16.392620 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:07:16.392642 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:07:16.393011 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:07:16.393020 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:07:16.394287 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:07:16.394292 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:07:16.396592 (+ 2300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:07:16.396608 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:07:16.398234 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:07:16.398239 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:07:16.398460 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:16.398496 (+ 36us) client_negotiation.cc:769] Sending connection context
0808 04:07:16.398590 (+ 94us) client_negotiation.cc:240] Negotiation successful
0808 04:07:16.398702 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1002,"mutex_wait_us":164,"thread_start_us":418,"threads_started":1}
I20260808 04:07:16.399487 27218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:16.391250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.2:45685 (local address 127.5.83.1:36007)
0808 04:07:16.391561 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0808 04:07:16.391567 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:07:16.391971 (+ 404us) server_negotiation.cc:407] Connection header received
0808 04:07:16.392160 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:07:16.392164 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:07:16.392227 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:07:16.392332 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:07:16.392937 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:16.394003 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:16.396777 (+ 2774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:16.398043 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:16.398155 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:16.398251 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0808 04:07:16.398751 (+ 500us) server_negotiation.cc:299] Negotiation successful
0808 04:07:16.398877 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260808 04:07:16.400089 27237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:16.390958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:36007 (local address 127.5.83.2:45685)
0808 04:07:16.391858 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:16.391875 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:07:16.392024 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:07:16.392375 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:07:16.392379 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:07:16.392405 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:07:16.392787 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:07:16.392799 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:07:16.394140 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:07:16.394143 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:07:16.396600 (+ 2457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:07:16.396609 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:07:16.398402 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:07:16.398407 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:07:16.398565 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:16.398598 (+ 33us) client_negotiation.cc:769] Sending connection context
0808 04:07:16.399587 (+ 989us) client_negotiation.cc:240] Negotiation successful
0808 04:07:16.399740 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":772,"mutex_wait_us":28,"thread_start_us":791,"threads_started":1}
I20260808 04:07:16.400130 27130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5eda73f1c4ef29f04019a07a67e7b" candidate_uuid: "37ad64cb60b94c699ebb75ac30c3865c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9f897aa131b4d91a4cde42bef14dc48" is_pre_election: true
I20260808 04:07:16.400671 27130 raft_consensus.cc:2454] T 3cd5eda73f1c4ef29f04019a07a67e7b P d9f897aa131b4d91a4cde42bef14dc48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37ad64cb60b94c699ebb75ac30c3865c in term 0.
I20260808 04:07:16.401302 26833 leader_election.cc:302] T 3cd5eda73f1c4ef29f04019a07a67e7b P 37ad64cb60b94c699ebb75ac30c3865c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37ad64cb60b94c699ebb75ac30c3865c, d9f897aa131b4d91a4cde42bef14dc48; no voters:
I20260808 04:07:16.401278 26723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5eda73f1c4ef29f04019a07a67e7b" candidate_uuid: "37ad64cb60b94c699ebb75ac30c3865c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ca4ca943204f63864c0587ad68d5e4" is_pre_election: true
I20260808 04:07:16.401652 27231 raft_consensus.cc:2781] T 3cd5eda73f1c4ef29f04019a07a67e7b P 37ad64cb60b94c699ebb75ac30c3865c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:07:16.401784 27231 raft_consensus.cc:482] T 3cd5eda73f1c4ef29f04019a07a67e7b P 37ad64cb60b94c699ebb75ac30c3865c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:07:16.401875 27231 raft_consensus.cc:3037] T 3cd5eda73f1c4ef29f04019a07a67e7b P 37ad64cb60b94c699ebb75ac30c3865c [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:07:16.401883 26723 raft_consensus.cc:2454] T 3cd5eda73f1c4ef29f04019a07a67e7b P 87ca4ca943204f63864c0587ad68d5e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37ad64cb60b94c699ebb75ac30c3865c in term 0.
I20260808 04:07:16.404202 27231 raft_consensus.cc:504] T 3cd5eda73f1c4ef29f04019a07a67e7b P 37ad64cb60b94c699ebb75ac30c3865c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ad64cb60b94c699ebb75ac30c3865c" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 45521 } } peers { permanent_uuid: "d9f897aa131b4d91a4cde42bef14dc48" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 36921 } } peers { permanent_uuid: "87ca4ca943204f63864c0587ad68d5e4" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 36007 } }
I20260808 04:07:16.404765 27231 leader_election.cc:288] T 3cd5eda73f1c4ef29f04019a07a67e7b P 37ad64cb60b94c699ebb75ac30c3865c [CANDIDATE]: Term 1 election: Requested vote from peers d9f897aa131b4d91a4cde42bef14dc48 (127.5.83.3:36921), 87ca4ca943204f63864c0587ad68d5e4 (127.5.83.1:36007)
I20260808 04:07:16.405107 27130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5eda73f1c4ef29f04019a07a67e7b" candidate_uuid: "37ad64cb60b94c699ebb75ac30c3865c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9f897aa131b4d91a4cde42bef14dc48"
I20260808 04:07:16.405198 26723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5eda73f1c4ef29f04019a07a67e7b" candidate_uuid: "37ad64cb60b94c699ebb75ac30c3865c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ca4ca943204f63864c0587ad68d5e4"
I20260808 04:07:16.405274 27130 raft_consensus.cc:3037] T 3cd5eda73f1c4ef29f04019a07a67e7b P d9f897aa131b4d91a4cde42bef14dc48 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:07:16.405375 26723 raft_consensus.cc:3037] T 3cd5eda73f1c4ef29f04019a07a67e7b P 87ca4ca943204f63864c0587ad68d5e4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:07:16.407377 27130 raft_consensus.cc:2454] T 3cd5eda73f1c4ef29f04019a07a67e7b P d9f897aa131b4d91a4cde42bef14dc48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37ad64cb60b94c699ebb75ac30c3865c in term 1.
I20260808 04:07:16.407529 26723 raft_consensus.cc:2454] T 3cd5eda73f1c4ef29f04019a07a67e7b P 87ca4ca943204f63864c0587ad68d5e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37ad64cb60b94c699ebb75ac30c3865c in term 1.
I20260808 04:07:16.407759 26833 leader_election.cc:302] T 3cd5eda73f1c4ef29f04019a07a67e7b P 37ad64cb60b94c699ebb75ac30c3865c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37ad64cb60b94c699ebb75ac30c3865c, d9f897aa131b4d91a4cde42bef14dc48; no voters:
I20260808 04:07:16.408033 27231 raft_consensus.cc:2781] T 3cd5eda73f1c4ef29f04019a07a67e7b P 37ad64cb60b94c699ebb75ac30c3865c [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:07:16.408453 27231 raft_consensus.cc:686] T 3cd5eda73f1c4ef29f04019a07a67e7b P 37ad64cb60b94c699ebb75ac30c3865c [term 1 LEADER]: Becoming Leader. State: Replica: 37ad64cb60b94c699ebb75ac30c3865c, State: Running, Role: LEADER
I20260808 04:07:16.408878 27231 consensus_queue.cc:237] T 3cd5eda73f1c4ef29f04019a07a67e7b P 37ad64cb60b94c699ebb75ac30c3865c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ad64cb60b94c699ebb75ac30c3865c" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 45521 } } peers { permanent_uuid: "d9f897aa131b4d91a4cde42bef14dc48" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 36921 } } peers { permanent_uuid: "87ca4ca943204f63864c0587ad68d5e4" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 36007 } }
I20260808 04:07:16.413249 26515 catalog_manager.cc:5495] T 3cd5eda73f1c4ef29f04019a07a67e7b P 37ad64cb60b94c699ebb75ac30c3865c reported cstate change: term changed from 0 to 1, leader changed from <none> to 37ad64cb60b94c699ebb75ac30c3865c (127.5.83.2). New cstate: current_term: 1 leader_uuid: "37ad64cb60b94c699ebb75ac30c3865c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ad64cb60b94c699ebb75ac30c3865c" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 45521 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d9f897aa131b4d91a4cde42bef14dc48" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 36921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87ca4ca943204f63864c0587ad68d5e4" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 36007 } health_report { overall_health: UNKNOWN } } }
I20260808 04:07:16.441857 27214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:16.436491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54040 (local address 127.5.83.2:45521)
0808 04:07:16.436710 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0808 04:07:16.436715 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:07:16.436734 (+ 19us) server_negotiation.cc:407] Connection header received
0808 04:07:16.436967 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:07:16.436971 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:07:16.437046 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:07:16.437169 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:07:16.437567 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:16.438287 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:16.439761 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:16.440180 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:16.440289 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:16.440800 (+ 511us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:07:16.441277 (+ 477us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:07:16.441408 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0808 04:07:16.441528 (+ 120us) server_negotiation.cc:299] Negotiation successful
0808 04:07:16.441608 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260808 04:07:16.449667 27130 raft_consensus.cc:1261] T 3cd5eda73f1c4ef29f04019a07a67e7b P d9f897aa131b4d91a4cde42bef14dc48 [term 1 FOLLOWER]: Refusing update from remote peer 37ad64cb60b94c699ebb75ac30c3865c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:07:16.449671 26723 raft_consensus.cc:1261] T 3cd5eda73f1c4ef29f04019a07a67e7b P 87ca4ca943204f63864c0587ad68d5e4 [term 1 FOLLOWER]: Refusing update from remote peer 37ad64cb60b94c699ebb75ac30c3865c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:07:16.450565 27231 consensus_queue.cc:1035] T 3cd5eda73f1c4ef29f04019a07a67e7b P 37ad64cb60b94c699ebb75ac30c3865c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9f897aa131b4d91a4cde42bef14dc48" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 36921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:07:16.450903 27239 consensus_queue.cc:1035] T 3cd5eda73f1c4ef29f04019a07a67e7b P 37ad64cb60b94c699ebb75ac30c3865c [LEADER]: Connected to new peer: Peer: permanent_uuid: "87ca4ca943204f63864c0587ad68d5e4" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 36007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 04:07:16.455765 27202 tablet.cc:2369] T 3cd5eda73f1c4ef29f04019a07a67e7b P d9f897aa131b4d91a4cde42bef14dc48: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:07:16.475356 26515 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:33370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:07:16.475608 26515 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:33370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:07:16.478621 26515 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 231726dbe3ec4fd49e58e8c2a800ed19: Sending DeleteTablet for 3 replicas of tablet 3cd5eda73f1c4ef29f04019a07a67e7b
I20260808 04:07:16.479967 27100 tablet_service.cc:1514] Processing DeleteTablet for tablet 3cd5eda73f1c4ef29f04019a07a67e7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:07:16 UTC) from {username='kudu', principal='kudu/127.5.83.62@KRBTEST.COM'} at 127.0.0.1:51896
I20260808 04:07:16.480000 26695 tablet_service.cc:1514] Processing DeleteTablet for tablet 3cd5eda73f1c4ef29f04019a07a67e7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:07:16 UTC) from {username='kudu', principal='kudu/127.5.83.62@KRBTEST.COM'} at 127.0.0.1:39838
I20260808 04:07:16.480159 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 26568
I20260808 04:07:16.479641 26907 tablet_service.cc:1514] Processing DeleteTablet for tablet 3cd5eda73f1c4ef29f04019a07a67e7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:07:16 UTC) from {username='kudu', principal='kudu/127.5.83.62@KRBTEST.COM'} at 127.0.0.1:54034
I20260808 04:07:16.488691 27258 tablet_replica.cc:330] T 3cd5eda73f1c4ef29f04019a07a67e7b P 37ad64cb60b94c699ebb75ac30c3865c: stopping tablet replica
I20260808 04:07:16.488858 27256 tablet_replica.cc:330] T 3cd5eda73f1c4ef29f04019a07a67e7b P d9f897aa131b4d91a4cde42bef14dc48: stopping tablet replica
I20260808 04:07:16.489115 27258 raft_consensus.cc:2229] T 3cd5eda73f1c4ef29f04019a07a67e7b P 37ad64cb60b94c699ebb75ac30c3865c [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:07:16.489212 27256 raft_consensus.cc:2229] T 3cd5eda73f1c4ef29f04019a07a67e7b P d9f897aa131b4d91a4cde42bef14dc48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:07:16.489634 27256 raft_consensus.cc:2258] T 3cd5eda73f1c4ef29f04019a07a67e7b P d9f897aa131b4d91a4cde42bef14dc48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:07:16.489892 27258 raft_consensus.cc:2258] T 3cd5eda73f1c4ef29f04019a07a67e7b P 37ad64cb60b94c699ebb75ac30c3865c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:07:16.490357 27257 tablet_replica.cc:330] T 3cd5eda73f1c4ef29f04019a07a67e7b P 87ca4ca943204f63864c0587ad68d5e4: stopping tablet replica
I20260808 04:07:16.490706 27257 raft_consensus.cc:2229] T 3cd5eda73f1c4ef29f04019a07a67e7b P 87ca4ca943204f63864c0587ad68d5e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:07:16.491151 27257 raft_consensus.cc:2258] T 3cd5eda73f1c4ef29f04019a07a67e7b P 87ca4ca943204f63864c0587ad68d5e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:07:16.492313 27256 ts_tablet_manager.cc:1905] T 3cd5eda73f1c4ef29f04019a07a67e7b P d9f897aa131b4d91a4cde42bef14dc48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:07:16.493251 27258 ts_tablet_manager.cc:1905] T 3cd5eda73f1c4ef29f04019a07a67e7b P 37ad64cb60b94c699ebb75ac30c3865c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:07:16.496833 27257 ts_tablet_manager.cc:1905] T 3cd5eda73f1c4ef29f04019a07a67e7b P 87ca4ca943204f63864c0587ad68d5e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:07:16.499533 27256 ts_tablet_manager.cc:1918] T 3cd5eda73f1c4ef29f04019a07a67e7b P d9f897aa131b4d91a4cde42bef14dc48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:07:16.499769 27256 log.cc:1196] T 3cd5eda73f1c4ef29f04019a07a67e7b P d9f897aa131b4d91a4cde42bef14dc48: Deleting WAL directory at /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-2/wal/wals/3cd5eda73f1c4ef29f04019a07a67e7b
I20260808 04:07:16.500376 27256 ts_tablet_manager.cc:1939] T 3cd5eda73f1c4ef29f04019a07a67e7b P d9f897aa131b4d91a4cde42bef14dc48: Deleting consensus metadata
I20260808 04:07:16.500428 27218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:16.481492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39844 (local address 127.5.83.1:36007)
0808 04:07:16.481652 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0808 04:07:16.481658 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:07:16.481676 (+ 18us) server_negotiation.cc:407] Connection header received
0808 04:07:16.481725 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:07:16.481730 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:07:16.481818 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:07:16.481930 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:07:16.483994 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:16.485128 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:16.487175 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:16.487845 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:16.487978 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:16.492735 (+ 4757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:07:16.492748 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:07:16.492763 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:07:16.492817 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:07:16.496468 (+ 3651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:16.497354 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:16.497362 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:16.497366 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:16.497512 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:16.497951 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:16.497956 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:16.497960 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:16.498408 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:07:16.498710 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0808 04:07:16.499874 (+ 1164us) server_negotiation.cc:299] Negotiation successful
0808 04:07:16.500083 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 04:07:16.501027 26792 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:07:16.502473 26493 catalog_manager.cc:4841] TS d9f897aa131b4d91a4cde42bef14dc48 (127.5.83.3:36921): tablet 3cd5eda73f1c4ef29f04019a07a67e7b (table test-table [id=4e89e46f1c02454d9b47d07a0df70816]) successfully deleted
I20260808 04:07:16.503474 27258 ts_tablet_manager.cc:1918] T 3cd5eda73f1c4ef29f04019a07a67e7b P 37ad64cb60b94c699ebb75ac30c3865c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:07:16.503652 27258 log.cc:1196] T 3cd5eda73f1c4ef29f04019a07a67e7b P 37ad64cb60b94c699ebb75ac30c3865c: Deleting WAL directory at /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-1/wal/wals/3cd5eda73f1c4ef29f04019a07a67e7b
I20260808 04:07:16.504249 27258 ts_tablet_manager.cc:1939] T 3cd5eda73f1c4ef29f04019a07a67e7b P 37ad64cb60b94c699ebb75ac30c3865c: Deleting consensus metadata
I20260808 04:07:16.505923 26494 catalog_manager.cc:4841] TS 37ad64cb60b94c699ebb75ac30c3865c (127.5.83.2:45521): tablet 3cd5eda73f1c4ef29f04019a07a67e7b (table test-table [id=4e89e46f1c02454d9b47d07a0df70816]) successfully deleted
I20260808 04:07:16.593901 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 26568
W20260808 04:07:16.616737 26491 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.83.1:36007 (error 108)
I20260808 04:07:16.617545 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 26803
I20260808 04:07:16.618829 27221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:16.618131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:36007 (local address 127.0.0.1:39852)
0808 04:07:16.618547 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:16.618663 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.1:36007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":284}
W20260808 04:07:16.619102 26491 catalog_manager.cc:4584] TS 87ca4ca943204f63864c0587ad68d5e4 (127.5.83.1:36007): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3cd5eda73f1c4ef29f04019a07a67e7b: Network error: Client connection negotiation failed: client connection to 127.5.83.1:36007: connect: Connection refused (error 111)
I20260808 04:07:16.631487 27214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:16.618284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54054 (local address 127.5.83.2:45521)
0808 04:07:16.618460 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0808 04:07:16.618467 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:07:16.618604 (+ 137us) server_negotiation.cc:407] Connection header received
0808 04:07:16.618801 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:07:16.618808 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:07:16.618886 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:07:16.618975 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:07:16.620496 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:16.621297 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:16.622537 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:16.622983 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:16.623095 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:16.625721 (+ 2626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:07:16.625735 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:07:16.625750 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:07:16.625785 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:07:16.628838 (+ 3053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:16.629445 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:16.629456 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:16.629458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:16.629603 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:16.629976 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:16.629979 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:16.629981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:16.630328 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:07:16.630650 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0808 04:07:16.630911 (+ 261us) server_negotiation.cc:299] Negotiation successful
0808 04:07:16.631126 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 04:07:16.632187 26995 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:07:16.672606 27221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:16.672163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:36007 (local address 127.0.0.1:39860)
0808 04:07:16.672348 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:16.672449 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.1:36007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260808 04:07:16.673659 27221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:16.673324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:36007 (local address 127.0.0.1:39876)
0808 04:07:16.673461 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:16.673546 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.1:36007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260808 04:07:16.734100 27221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:16.733621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:36007 (local address 127.0.0.1:39880)
0808 04:07:16.733824 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:16.733928 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.1:36007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260808 04:07:16.735117 27221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:16.734783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:36007 (local address 127.0.0.1:39892)
0808 04:07:16.734921 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:16.735007 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.1:36007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260808 04:07:16.743680 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 26803
I20260808 04:07:16.765477 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 27007
I20260808 04:07:16.778081 27216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:16.766196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51904 (local address 127.5.83.3:36921)
0808 04:07:16.766391 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0808 04:07:16.766396 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:07:16.766416 (+ 20us) server_negotiation.cc:407] Connection header received
0808 04:07:16.766616 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:07:16.766620 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:07:16.766692 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:07:16.766776 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:07:16.768053 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:16.768772 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:16.770027 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:16.770400 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:16.770484 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:16.773099 (+ 2615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:07:16.773114 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:07:16.773130 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:07:16.773166 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:07:16.775836 (+ 2670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:16.776404 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:16.776411 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:16.776413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:16.776550 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:16.776827 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:16.776830 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:16.776832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:16.777167 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:07:16.777373 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0808 04:07:16.777595 (+ 222us) server_negotiation.cc:299] Negotiation successful
0808 04:07:16.777764 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260808 04:07:16.778668 27191 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:07:16.815766 27221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:16.815154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:36007 (local address 127.0.0.1:39902)
0808 04:07:16.815401 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:16.815534 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.1:36007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260808 04:07:16.816840 27221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:16.816408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:36007 (local address 127.0.0.1:39908)
0808 04:07:16.816611 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:16.816712 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.1:36007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260808 04:07:16.866627 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 27007
I20260808 04:07:16.887599 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 26465
I20260808 04:07:16.888767 26560 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:07:17.004814 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 26465
I20260808 04:07:17.034065 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:35721
--webserver_interface=127.5.83.62
--webserver_port=34815
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab
--principal=kudu/127.5.83.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:39647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.62:35721
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:07:17.375499 27266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:07:17.375811 27266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:07:17.375927 27266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:07:17.382884 27266 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:07:17.382978 27266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:07:17.383030 27266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:07:17.383085 27266 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:07:17.383127 27266 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:07:17.392591 27266 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:39647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.62:35721
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab
--principal=kudu/127.5.83.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:35721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.62
--webserver_port=34815
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:07:17.393805 27266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:07:17.395017 27266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:07:17.405951 27271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:07:17.405928 27272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:07:17.405895 27275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:07:17.406934 27266 server_base.cc:1034] running on GCE node
I20260808 04:07:17.407612 27266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:07:17.408931 27266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:07:17.410132 27266 hybrid_clock.cc:648] HybridClock initialized: now 1786162037410071 us; error 46 us; skew 500 ppm
I20260808 04:07:17.410444 27266 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:07:17 dist-test-slave-1l3l krb5kdc[25636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162037, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:07:17.413347 27266 init.cc:377] Logged in from keytab as kudu/127.5.83.62@KRBTEST.COM (short username kudu)
I20260808 04:07:17.414672 27266 webserver.cc:460] Webserver started at http://127.5.83.62:34815/ using document root <none> and password file <none>
I20260808 04:07:17.415359 27266 fs_manager.cc:359] Metadata directory not provided
I20260808 04:07:17.415475 27266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:07:17.420614 27266 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:07:17.423328 27287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:07:17.424575 27266 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:07:17.424731 27266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/master-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/master-0/wal
uuid: "231726dbe3ec4fd49e58e8c2a800ed19"
format_stamp: "Formatted at 2026-08-08 04:07:11 on dist-test-slave-1l3l"
server_key: "1c35a255f052a90bdb28ea785819b122"
server_key_iv: "798b3c5a5fe4e2a133f9d9ba66eb5ecd"
server_key_version: "encryptionkey@0"
I20260808 04:07:17.425633 27266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:07:17.459671 27266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:07:17.463779 27266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:07:17.464046 27266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:07:17.481312 27266 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.62:35721
I20260808 04:07:17.481353 27366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.62:35721 every 8 connection(s)
I20260808 04:07:17.482686 27266 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
I20260808 04:07:17.489416 27367 sys_catalog.cc:263] Verifying existing consensus state
I20260808 04:07:17.489603 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 27266
I20260808 04:07:17.490556 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.1:36007
--local_ip_for_outbound_sockets=127.5.83.1
--tserver_master_addrs=127.5.83.62:35721
--webserver_port=37729
--webserver_interface=127.5.83.1
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.1.keytab
--principal=kudu/127.5.83.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:39647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:07:17.493805 27367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 231726dbe3ec4fd49e58e8c2a800ed19: Bootstrap starting.
I20260808 04:07:17.515744 27367 log.cc:824] T 00000000000000000000000000000000 P 231726dbe3ec4fd49e58e8c2a800ed19: Log is configured to *not* fsync() on all Append() calls
I20260808 04:07:17.532085 27367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 231726dbe3ec4fd49e58e8c2a800ed19: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 04:07:17.532573 27367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 231726dbe3ec4fd49e58e8c2a800ed19: Bootstrap complete.
I20260808 04:07:17.541404 27367 raft_consensus.cc:348] T 00000000000000000000000000000000 P 231726dbe3ec4fd49e58e8c2a800ed19 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "231726dbe3ec4fd49e58e8c2a800ed19" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 35721 } }
I20260808 04:07:17.542411 27367 raft_consensus.cc:729] T 00000000000000000000000000000000 P 231726dbe3ec4fd49e58e8c2a800ed19 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 231726dbe3ec4fd49e58e8c2a800ed19, State: Initialized, Role: FOLLOWER
I20260808 04:07:17.542972 27367 consensus_queue.cc:260] T 00000000000000000000000000000000 P 231726dbe3ec4fd49e58e8c2a800ed19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "231726dbe3ec4fd49e58e8c2a800ed19" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 35721 } }
I20260808 04:07:17.543176 27367 raft_consensus.cc:388] T 00000000000000000000000000000000 P 231726dbe3ec4fd49e58e8c2a800ed19 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:07:17.543282 27367 raft_consensus.cc:482] T 00000000000000000000000000000000 P 231726dbe3ec4fd49e58e8c2a800ed19 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:07:17.543462 27367 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 231726dbe3ec4fd49e58e8c2a800ed19 [term 2 FOLLOWER]: Advancing to term 3
I20260808 04:07:17.547214 27367 raft_consensus.cc:504] T 00000000000000000000000000000000 P 231726dbe3ec4fd49e58e8c2a800ed19 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "231726dbe3ec4fd49e58e8c2a800ed19" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 35721 } }
I20260808 04:07:17.547676 27367 leader_election.cc:302] T 00000000000000000000000000000000 P 231726dbe3ec4fd49e58e8c2a800ed19 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 231726dbe3ec4fd49e58e8c2a800ed19; no voters:
I20260808 04:07:17.548331 27367 leader_election.cc:288] T 00000000000000000000000000000000 P 231726dbe3ec4fd49e58e8c2a800ed19 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260808 04:07:17.548454 27371 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 231726dbe3ec4fd49e58e8c2a800ed19 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 04:07:17.549607 27371 raft_consensus.cc:686] T 00000000000000000000000000000000 P 231726dbe3ec4fd49e58e8c2a800ed19 [term 3 LEADER]: Becoming Leader. State: Replica: 231726dbe3ec4fd49e58e8c2a800ed19, State: Running, Role: LEADER
I20260808 04:07:17.550209 27371 consensus_queue.cc:237] T 00000000000000000000000000000000 P 231726dbe3ec4fd49e58e8c2a800ed19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "231726dbe3ec4fd49e58e8c2a800ed19" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 35721 } }
I20260808 04:07:17.550539 27367 sys_catalog.cc:564] T 00000000000000000000000000000000 P 231726dbe3ec4fd49e58e8c2a800ed19 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:07:17.552932 27374 sys_catalog.cc:455] T 00000000000000000000000000000000 P 231726dbe3ec4fd49e58e8c2a800ed19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 231726dbe3ec4fd49e58e8c2a800ed19. Latest consensus state: current_term: 3 leader_uuid: "231726dbe3ec4fd49e58e8c2a800ed19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "231726dbe3ec4fd49e58e8c2a800ed19" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 35721 } } }
I20260808 04:07:17.553331 27374 sys_catalog.cc:458] T 00000000000000000000000000000000 P 231726dbe3ec4fd49e58e8c2a800ed19 [sys.catalog]: This master's current role is: LEADER
I20260808 04:07:17.552901 27372 sys_catalog.cc:455] T 00000000000000000000000000000000 P 231726dbe3ec4fd49e58e8c2a800ed19 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "231726dbe3ec4fd49e58e8c2a800ed19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "231726dbe3ec4fd49e58e8c2a800ed19" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 35721 } } }
I20260808 04:07:17.553635 27372 sys_catalog.cc:458] T 00000000000000000000000000000000 P 231726dbe3ec4fd49e58e8c2a800ed19 [sys.catalog]: This master's current role is: LEADER
I20260808 04:07:17.554421 27382 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:07:17.570330 27382 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:07:17.572427 27382 catalog_manager.cc:1259] Loaded cluster ID: f2cf49dc4abb44b39ec9cf72bb35b00e
I20260808 04:07:17.572599 27382 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260808 04:07:17.574707 27382 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f846de799ad google::LogMessage::Fail() at ??:0
@ 0x7f846de7c88a google::LogMessage::SendToLog() at ??:0
@ 0x7f846de794de google::LogMessage::Flush() at ??:0
@ 0x7f846de7aa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f84838ac512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f84838a9976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f8483931610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
I20260808 04:07:17.625419 27406 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7f8483931439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f8470ac5345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f8470ac891c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f8470ac8799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f8470aa3eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f8473d7d6db start_thread at ??:0
@ 0x7f846cd1361f clone at ??:0
W20260808 04:07:17.854713 27369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:07:17.854983 27369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:07:17.855111 27369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:07:17.862243 27369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:07:17.862357 27369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:07:17.862522 27369 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.1
I20260808 04:07:17.872021 27369 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:39647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.1.keytab
--principal=kudu/127.5.83.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.1:36007
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.83.1
--webserver_port=37729
--tserver_master_addrs=127.5.83.62:35721
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.1
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:07:17.873209 27369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:07:17.874511 27369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:07:17.887006 27429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:07:17.887055 27426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:07:17.887077 27427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:07:17.887905 27369 server_base.cc:1034] running on GCE node
I20260808 04:07:17.888495 27369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:07:17.889359 27369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:07:17.890564 27369 hybrid_clock.cc:648] HybridClock initialized: now 1786162037890508 us; error 51 us; skew 500 ppm
I20260808 04:07:17.890849 27369 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:07:17 dist-test-slave-1l3l krb5kdc[25636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162037, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:07:17.893806 27369 init.cc:377] Logged in from keytab as kudu/127.5.83.1@KRBTEST.COM (short username kudu)
I20260808 04:07:17.895156 27369 webserver.cc:460] Webserver started at http://127.5.83.1:37729/ using document root <none> and password file <none>
I20260808 04:07:17.895869 27369 fs_manager.cc:359] Metadata directory not provided
I20260808 04:07:17.895995 27369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:07:17.900988 27369 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260808 04:07:17.903831 27439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:07:17.905022 27369 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 04:07:17.905172 27369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-0/wal
uuid: "87ca4ca943204f63864c0587ad68d5e4"
format_stamp: "Formatted at 2026-08-08 04:07:12 on dist-test-slave-1l3l"
server_key: "6003d2fd8dc22147facf5cada69e4412"
server_key_iv: "2193ebbc6e935dc13dc2eb0474e0a27f"
server_key_version: "encryptionkey@0"
I20260808 04:07:17.906070 27369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:07:17.921125 27369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:07:17.924616 27369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:07:17.924849 27369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:07:17.925714 27369 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:07:17.927366 27450 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260808 04:07:17.932327 27450 ts_tablet_manager.cc:1852] T 3cd5eda73f1c4ef29f04019a07a67e7b P 87ca4ca943204f63864c0587ad68d5e4: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260808 04:07:17.933377 27450 ts_tablet_manager.cc:1905] T 3cd5eda73f1c4ef29f04019a07a67e7b P 87ca4ca943204f63864c0587ad68d5e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:07:17.941232 27450 ts_tablet_manager.cc:1918] T 3cd5eda73f1c4ef29f04019a07a67e7b P 87ca4ca943204f63864c0587ad68d5e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:07:17.941431 27450 log.cc:1196] T 3cd5eda73f1c4ef29f04019a07a67e7b P 87ca4ca943204f63864c0587ad68d5e4: Deleting WAL directory at /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-0/wal/wals/3cd5eda73f1c4ef29f04019a07a67e7b
I20260808 04:07:17.941869 27450 ts_tablet_manager.cc:1939] T 3cd5eda73f1c4ef29f04019a07a67e7b P 87ca4ca943204f63864c0587ad68d5e4: Deleting consensus metadata
I20260808 04:07:17.942229 27369 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260808 04:07:17.942333 27369 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.015s user 0.001s sys 0.000s
I20260808 04:07:17.942462 27369 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:07:17.942523 27369 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:07:17.974196 27369 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.1:36007
I20260808 04:07:17.974259 27621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.1:36007 every 8 connection(s)
I20260808 04:07:17.975495 27369 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
I20260808 04:07:17.976383 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 27369
I20260808 04:07:17.977095 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.2:45521
--local_ip_for_outbound_sockets=127.5.83.2
--tserver_master_addrs=127.5.83.62:35721
--webserver_port=41049
--webserver_interface=127.5.83.2
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.2.keytab
--principal=kudu/127.5.83.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:39647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:07:17.980592 27625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:17.978425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:35721 (local address 127.5.83.1:53539)
0808 04:07:17.979232 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:17.979482 (+ 250us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:35721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":312,"threads_started":1}
W20260808 04:07:17.981210 27622 heartbeater.cc:644] Failed to heartbeat to 127.5.83.62:35721 (0 consecutive failures): Network error: Failed to ping master at 127.5.83.62:35721: Client connection negotiation failed: client connection to 127.5.83.62:35721: connect: Connection refused (error 111)
I20260808 04:07:17.982026 27625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:17.981675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:35721 (local address 127.5.83.1:37385)
0808 04:07:17.981829 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:17.981909 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:35721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260808 04:07:17.982911 27625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:17.982565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:35721 (local address 127.5.83.1:32857)
0808 04:07:17.982698 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:17.982777 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:35721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260808 04:07:17.983139 27622 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 04:07:18.297556 27626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:07:18.297814 27626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:07:18.297950 27626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:07:18.304904 27626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:07:18.304997 27626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:07:18.305143 27626 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.2
I20260808 04:07:18.314427 27626 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:39647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.2.keytab
--principal=kudu/127.5.83.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.2:45521
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.83.2
--webserver_port=41049
--tserver_master_addrs=127.5.83.62:35721
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.2
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:07:18.315604 27626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:07:18.316900 27626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:07:18.329169 27632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:07:18.329172 27633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:07:18.329205 27638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:07:18.330017 27626 server_base.cc:1034] running on GCE node
I20260808 04:07:18.330559 27626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:07:18.331398 27626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:07:18.332608 27626 hybrid_clock.cc:648] HybridClock initialized: now 1786162038332553 us; error 44 us; skew 500 ppm
I20260808 04:07:18.332898 27626 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:07:18 dist-test-slave-1l3l krb5kdc[25636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162038, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:07:18.335837 27626 init.cc:377] Logged in from keytab as kudu/127.5.83.2@KRBTEST.COM (short username kudu)
I20260808 04:07:18.337122 27626 webserver.cc:460] Webserver started at http://127.5.83.2:41049/ using document root <none> and password file <none>
I20260808 04:07:18.337735 27626 fs_manager.cc:359] Metadata directory not provided
I20260808 04:07:18.337847 27626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:07:18.342751 27626 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:07:18.345245 27650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:07:18.346477 27626 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 04:07:18.346624 27626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-1/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-1/wal
uuid: "37ad64cb60b94c699ebb75ac30c3865c"
format_stamp: "Formatted at 2026-08-08 04:07:13 on dist-test-slave-1l3l"
server_key: "04f99e3d10123a3a0e3abe6a15c18711"
server_key_iv: "423165bcd7fceddee3de5393e2d68096"
server_key_version: "encryptionkey@0"
I20260808 04:07:18.347517 27626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:07:18.378376 27626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:07:18.382129 27626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:07:18.382387 27626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:07:18.383312 27626 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:07:18.384603 27626 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:07:18.384709 27626 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:07:18.384811 27626 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:07:18.384886 27626 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:07:18.417301 27626 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.2:45521
I20260808 04:07:18.417369 27820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.2:45521 every 8 connection(s)
I20260808 04:07:18.418646 27626 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
I20260808 04:07:18.422475 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 27626
I20260808 04:07:18.423256 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.3:36921
--local_ip_for_outbound_sockets=127.5.83.3
--tserver_master_addrs=127.5.83.62:35721
--webserver_port=46075
--webserver_interface=127.5.83.3
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.3.keytab
--principal=kudu/127.5.83.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:39647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:07:18.423815 27827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:18.421532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:35721 (local address 127.5.83.2:53149)
0808 04:07:18.422346 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:18.422604 (+ 258us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:35721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":279,"threads_started":1}
W20260808 04:07:18.424521 27821 heartbeater.cc:644] Failed to heartbeat to 127.5.83.62:35721 (0 consecutive failures): Network error: Failed to ping master at 127.5.83.62:35721: Client connection negotiation failed: client connection to 127.5.83.62:35721: connect: Connection refused (error 111)
I20260808 04:07:18.425366 27827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:18.425002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:35721 (local address 127.5.83.2:46975)
0808 04:07:18.425147 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:18.425225 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:35721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260808 04:07:18.426195 27827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:18.425860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:35721 (local address 127.5.83.2:54893)
0808 04:07:18.425987 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:18.426073 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:35721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260808 04:07:18.426412 27821 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 04:07:18.746934 27829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:07:18.747192 27829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:07:18.747310 27829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:07:18.754485 27829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:07:18.754590 27829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:07:18.754729 27829 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.3
I20260808 04:07:18.764303 27829 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:39647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.3.keytab
--principal=kudu/127.5.83.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.3:36921
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.83.3
--webserver_port=46075
--tserver_master_addrs=127.5.83.62:35721
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.3
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:07:18.765522 27829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:07:18.766784 27829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:07:18.779582 27836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:07:18.779579 27838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:07:18.779764 27840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:07:18.780690 27829 server_base.cc:1034] running on GCE node
I20260808 04:07:18.781229 27829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:07:18.782084 27829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:07:18.783268 27829 hybrid_clock.cc:648] HybridClock initialized: now 1786162038783232 us; error 31 us; skew 500 ppm
I20260808 04:07:18.783598 27829 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:07:18 dist-test-slave-1l3l krb5kdc[25636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162038, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:07:18.786639 27829 init.cc:377] Logged in from keytab as kudu/127.5.83.3@KRBTEST.COM (short username kudu)
I20260808 04:07:18.788044 27829 webserver.cc:460] Webserver started at http://127.5.83.3:46075/ using document root <none> and password file <none>
I20260808 04:07:18.788681 27829 fs_manager.cc:359] Metadata directory not provided
I20260808 04:07:18.788797 27829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:07:18.793983 27829 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 04:07:18.796617 27852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:07:18.797895 27829 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:07:18.798045 27829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-2/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-2/wal
uuid: "d9f897aa131b4d91a4cde42bef14dc48"
format_stamp: "Formatted at 2026-08-08 04:07:13 on dist-test-slave-1l3l"
server_key: "917e5a9e738808ecfa06cc7e9cbe0ddc"
server_key_iv: "ba4ec26d2ac5cec225a0c6ec200e4d02"
server_key_version: "encryptionkey@0"
I20260808 04:07:18.798978 27829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:07:18.811954 27829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:07:18.815562 27829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:07:18.815835 27829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:07:18.816716 27829 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:07:18.817946 27829 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:07:18.818041 27829 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:07:18.818146 27829 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:07:18.818216 27829 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:07:18.848539 27829 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.3:36921
I20260808 04:07:18.848595 28027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.3:36921 every 8 connection(s)
I20260808 04:07:18.849936 27829 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
I20260808 04:07:18.854578 28033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:18.852855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:35721 (local address 127.5.83.3:60167)
0808 04:07:18.853554 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:18.853750 (+ 196us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:35721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":263,"threads_started":1}
W20260808 04:07:18.855206 28028 heartbeater.cc:644] Failed to heartbeat to 127.5.83.62:35721 (0 consecutive failures): Network error: Failed to ping master at 127.5.83.62:35721: Client connection negotiation failed: client connection to 127.5.83.62:35721: connect: Connection refused (error 111)
I20260808 04:07:18.856019 28033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:18.855670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:35721 (local address 127.5.83.3:51709)
0808 04:07:18.855837 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:18.855912 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:35721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260808 04:07:18.856787 28033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:18.856498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:35721 (local address 127.5.83.3:43765)
0808 04:07:18.856610 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:18.856681 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:35721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260808 04:07:18.856998 28028 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260808 04:07:18.858675 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 27829
I20260808 04:07:18.984422 28037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:18.983640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:35721 (local address 127.5.83.1:57269)
0808 04:07:18.984098 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:18.984181 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:35721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":243,"threads_started":1}
I20260808 04:07:19.427755 28039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:19.426939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:35721 (local address 127.5.83.2:37561)
0808 04:07:19.427399 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:19.427501 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:35721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":211,"threads_started":1}
I20260808 04:07:19.858224 28040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:19.857481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:35721 (local address 127.5.83.3:58109)
0808 04:07:19.857915 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:19.857995 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:35721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":205,"threads_started":1}
I20260808 04:07:19.985961 28042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:19.985216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:35721 (local address 127.5.83.1:48463)
0808 04:07:19.985668 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:19.985762 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:35721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":243,"threads_started":1}
I20260808 04:07:20.429314 28043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:20.428591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:35721 (local address 127.5.83.2:37555)
0808 04:07:20.429080 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:20.429156 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:35721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":253,"threads_started":1}
I20260808 04:07:20.859792 28044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:20.859022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:35721 (local address 127.5.83.3:36621)
0808 04:07:20.859486 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:20.859577 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:35721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":226,"threads_started":1}
I20260808 04:07:20.987465 28045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:20.986764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:35721 (local address 127.5.83.1:39683)
0808 04:07:20.987207 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:20.987287 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:35721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":226,"threads_started":1}
I20260808 04:07:21.430856 28046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:21.430147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:35721 (local address 127.5.83.2:40305)
0808 04:07:21.430607 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:21.430685 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:35721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":261,"threads_started":1}
I20260808 04:07:21.861219 28048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:21.860556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:35721 (local address 127.5.83.3:57407)
0808 04:07:21.860976 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:21.861052 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:35721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":212,"threads_started":1}
I20260808 04:07:21.989115 28049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:21.988336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:35721 (local address 127.5.83.1:42381)
0808 04:07:21.988805 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:21.988896 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:35721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":229,"threads_started":1}
I20260808 04:07:22.432407 28050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:22.431685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:35721 (local address 127.5.83.2:37505)
0808 04:07:22.432156 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:22.432234 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:35721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":226,"threads_started":1}
I20260808 04:07:22.862758 28054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:22.862036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:35721 (local address 127.5.83.3:37611)
0808 04:07:22.862482 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:22.862561 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:35721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":222,"threads_started":1}
I20260808 04:07:22.990622 28060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:22.989933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:35721 (local address 127.5.83.1:41699)
0808 04:07:22.990385 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:22.990462 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:35721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":205,"threads_started":1}
I20260808 04:07:23.433915 28062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:23.433199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:35721 (local address 127.5.83.2:33217)
0808 04:07:23.433656 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:23.433735 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:35721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":247,"threads_started":1}
I20260808 04:07:23.864329 28063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:23.863612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:35721 (local address 127.5.83.3:38143)
0808 04:07:23.864085 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:23.864161 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:35721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":261,"threads_started":1}
I20260808 04:07:23.992094 28065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:23.991384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:35721 (local address 127.5.83.1:46937)
0808 04:07:23.991850 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:23.991935 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:35721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":222,"threads_started":1}
I20260808 04:07:24.435595 28066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:24.434824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:35721 (local address 127.5.83.2:40659)
0808 04:07:24.435327 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:24.435432 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:35721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":260,"threads_started":1}
I20260808 04:07:24.865836 28070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:24.865152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:35721 (local address 127.5.83.3:59659)
0808 04:07:24.865585 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:24.865670 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:35721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":228,"threads_started":1}
I20260808 04:07:24.993640 28072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:24.992846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:35721 (local address 127.5.83.1:40031)
0808 04:07:24.993352 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:24.993435 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:35721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":218,"threads_started":1}
I20260808 04:07:25.437280 28076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:25.436480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:35721 (local address 127.5.83.2:37085)
0808 04:07:25.436984 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:25.437074 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:35721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":215,"threads_started":1}
I20260808 04:07:25.867533 28077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:25.866729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:35721 (local address 127.5.83.3:45333)
0808 04:07:25.867219 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:25.867300 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:35721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":231,"threads_started":1}
I20260808 04:07:25.995218 28078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:25.994503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:35721 (local address 127.5.83.1:33761)
0808 04:07:25.994976 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:25.995054 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:35721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":248,"threads_started":1}
I20260808 04:07:26.438874 28080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:26.438153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:35721 (local address 127.5.83.2:44451)
0808 04:07:26.438622 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:26.438708 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:35721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":235,"threads_started":1}
I20260808 04:07:26.869115 28081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:26.868433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:35721 (local address 127.5.83.3:52445)
0808 04:07:26.868865 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:26.868952 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:35721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":222,"threads_started":1}
I20260808 04:07:26.999594 28082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:26.996127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:35721 (local address 127.5.83.1:42641)
0808 04:07:26.999295 (+ 3168us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:26.999399 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:35721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3064,"threads_started":1}
I20260808 04:07:27.440451 28083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:27.439673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:35721 (local address 127.5.83.2:52023)
0808 04:07:27.440197 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:27.440275 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:35721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":256,"threads_started":1}
I20260808 04:07:27.870657 28084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:27.869938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:35721 (local address 127.5.83.3:37261)
0808 04:07:27.870390 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:27.870487 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:35721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":217,"threads_started":1}
I20260808 04:07:28.001176 28087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:28.000498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:35721 (local address 127.5.83.1:55353)
0808 04:07:28.000936 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:28.001012 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:35721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":225,"threads_started":1}
I20260808 04:07:28.441972 28088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:28.441273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:35721 (local address 127.5.83.2:54051)
0808 04:07:28.441724 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:28.441803 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:35721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":253,"threads_started":1}
I20260808 04:07:28.872226 28089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:28.871485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:35721 (local address 127.5.83.3:60477)
0808 04:07:28.871984 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:28.872063 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:35721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":300,"threads_started":1}
I20260808 04:07:29.002720 28090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:29.002029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:35721 (local address 127.5.83.1:46171)
0808 04:07:29.002462 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:29.002539 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:35721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":242,"threads_started":1}
I20260808 04:07:29.443565 28093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:29.442831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:35721 (local address 127.5.83.2:57069)
0808 04:07:29.443264 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:29.443382 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:35721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":220,"threads_started":1}
I20260808 04:07:29.873853 28094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:29.873099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:35721 (local address 127.5.83.3:35051)
0808 04:07:29.873586 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:29.873678 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:35721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":250,"threads_started":1}
I20260808 04:07:30.004350 28095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:30.003612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:35721 (local address 127.5.83.1:46523)
0808 04:07:30.004109 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:30.004188 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:35721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":288,"threads_started":1}
I20260808 04:07:30.445096 28096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:30.444421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:35721 (local address 127.5.83.2:42823)
0808 04:07:30.444859 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:30.444937 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:35721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":234,"threads_started":1}
I20260808 04:07:30.875466 28098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:30.874708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:35721 (local address 127.5.83.3:49689)
0808 04:07:30.875190 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:30.875272 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:35721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":254,"threads_started":1}
I20260808 04:07:31.005941 28099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:31.005183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:35721 (local address 127.5.83.1:42217)
0808 04:07:31.005651 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:31.005731 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:35721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":234,"threads_started":1}
I20260808 04:07:31.446738 28100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:31.445959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:35721 (local address 127.5.83.2:40067)
0808 04:07:31.446468 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:31.446551 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:35721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":248,"threads_started":1}
I20260808 04:07:31.877142 28101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:31.876308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:35721 (local address 127.5.83.3:36473)
0808 04:07:31.876822 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:31.876907 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:35721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":258,"threads_started":1}
I20260808 04:07:32.007614 28102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:32.006835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:35721 (local address 127.5.83.1:37713)
0808 04:07:32.007335 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:32.007430 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:35721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":255,"threads_started":1}
I20260808 04:07:32.448302 28103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:32.447575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:35721 (local address 127.5.83.2:50055)
0808 04:07:32.448037 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:32.448118 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:35721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":244,"threads_started":1}
I20260808 04:07:32.878707 28104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:32.877972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:35721 (local address 127.5.83.3:56481)
0808 04:07:32.878439 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:32.878517 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:35721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":218,"threads_started":1}
I20260808 04:07:33.009429 28106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:33.008679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:35721 (local address 127.5.83.1:37617)
0808 04:07:33.009127 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:33.009206 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:35721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":224,"threads_started":1}
I20260808 04:07:33.450066 28107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:33.449332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:35721 (local address 127.5.83.2:56455)
0808 04:07:33.449794 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:33.449883 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:35721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":240,"threads_started":1}
I20260808 04:07:33.871153 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 27369
I20260808 04:07:33.880496 28110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:33.879785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:35721 (local address 127.5.83.3:47781)
0808 04:07:33.880198 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:33.880286 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:35721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":206,"threads_started":1}
I20260808 04:07:33.892333 28108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:33.872596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50560 (local address 127.5.83.1:36007)
0808 04:07:33.873094 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0808 04:07:33.873109 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 04:07:33.873147 (+ 38us) server_negotiation.cc:407] Connection header received
0808 04:07:33.874753 (+ 1606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:07:33.874762 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:07:33.875174 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:07:33.875475 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:07:33.876760 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:33.878082 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:33.879347 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:33.880321 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:33.880492 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:33.886313 (+ 5821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:07:33.886321 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:07:33.886336 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:07:33.886373 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:07:33.889124 (+ 2751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:33.889807 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:33.889813 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:33.889833 (+ 20us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:33.889998 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:33.890366 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:33.890371 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:33.890373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:33.890888 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:07:33.891171 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0808 04:07:33.891605 (+ 434us) server_negotiation.cc:299] Negotiation successful
0808 04:07:33.892050 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":206,"threads_started":1}
I20260808 04:07:33.893594 27615 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:07:33.977221 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 27369
I20260808 04:07:33.996398 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 27626
I20260808 04:07:34.013499 28113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:33.997462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53118 (local address 127.5.83.2:45521)
0808 04:07:33.998073 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0808 04:07:33.998088 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 04:07:33.998127 (+ 39us) server_negotiation.cc:407] Connection header received
0808 04:07:33.999641 (+ 1514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:07:33.999650 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:07:34.000158 (+ 508us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:07:34.000437 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:07:34.001844 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:34.003198 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:34.004529 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:34.005184 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:34.005307 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:34.007838 (+ 2531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:07:34.007846 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:07:34.007860 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:07:34.007916 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:07:34.010659 (+ 2743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:34.011289 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:34.011294 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:34.011301 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:34.011461 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:34.011778 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:34.011780 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:34.011782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:34.012265 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:07:34.012484 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0808 04:07:34.012812 (+ 328us) server_negotiation.cc:299] Negotiation successful
0808 04:07:34.013210 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":303,"threads_started":1}
I20260808 04:07:34.014837 27815 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:07:34.095743 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 27626
I20260808 04:07:34.117185 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 27829
I20260808 04:07:34.134918 28117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:34.118108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46794 (local address 127.5.83.3:36921)
0808 04:07:34.118728 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0808 04:07:34.118741 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 04:07:34.118783 (+ 42us) server_negotiation.cc:407] Connection header received
0808 04:07:34.120464 (+ 1681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:07:34.120474 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:07:34.120888 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:07:34.121171 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:07:34.122748 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:34.124116 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:34.125632 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:34.126284 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:34.126419 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:34.129190 (+ 2771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:07:34.129197 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:07:34.129210 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:07:34.129246 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:07:34.132010 (+ 2764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:34.132667 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:34.132672 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:34.132678 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:34.132839 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:34.133164 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:34.133167 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:34.133169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:34.133640 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:07:34.133861 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0808 04:07:34.134235 (+ 374us) server_negotiation.cc:299] Negotiation successful
0808 04:07:34.134604 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":301,"threads_started":1}
I20260808 04:07:34.136255 28022 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:07:34.218169 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 27829
2026-08-08T04:07:34Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24081 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:07:34 dist-test-slave-1l3l krb5kdc[28123](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:07:34 dist-test-slave-1l3l krb5kdc[28123](info): set up 2 sockets
Aug 08 04:07:34 dist-test-slave-1l3l krb5kdc[28123](info): commencing operation
krb5kdc: starting...
W20260808 04:07:35.412633 5452 mini_kdc.cc:121] Time spent starting KDC: real 1.151s user 0.005s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/krb5kdc/test-user.keytab.
Aug 08 04:07:35 dist-test-slave-1l3l krb5kdc[28123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162055, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:07:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:07:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.83.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab.
Entry for principal kudu/127.5.83.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab.
I20260808 04:07:35.655499 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:38749
--webserver_interface=127.5.83.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab
--principal=kudu/127.5.83.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:36095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.62:38749
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:07:35.974803 28146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:07:35.975076 28146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:07:35.975204 28146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:07:35.982275 28146 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260808 04:07:35.982365 28146 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260808 04:07:35.982409 28146 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260808 04:07:35.982479 28146 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:07:35.982543 28146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:07:35.982636 28146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:07:35.982690 28146 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:07:35.982774 28146 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:07:35.993466 28146 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:36095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.5.83.62:38749
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab
--principal=kudu/127.5.83.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:38749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:07:35.994720 28146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:07:35.996073 28146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:07:36.007278 28157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:07:36.007308 28155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:07:36.007599 28154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:07:36.008352 28146 server_base.cc:1034] running on GCE node
I20260808 04:07:36.009097 28146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:07:36.010349 28146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:07:36.011557 28146 hybrid_clock.cc:648] HybridClock initialized: now 1786162056011497 us; error 35 us; skew 500 ppm
I20260808 04:07:36.011888 28146 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:07:36 dist-test-slave-1l3l krb5kdc[28123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162056, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:07:36.014770 28146 init.cc:377] Logged in from keytab as kudu/127.5.83.62@KRBTEST.COM (short username kudu)
I20260808 04:07:36.016113 28146 webserver.cc:460] Webserver started at http://127.5.83.62:39963/ using document root <none> and password file <none>
I20260808 04:07:36.016739 28146 fs_manager.cc:359] Metadata directory not provided
I20260808 04:07:36.016825 28146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:07:36.017081 28146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:07:36.019896 28146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/master-0/data/instance:
uuid: "ce0c76f0336e4437aa940b8e18eedbab"
format_stamp: "Formatted at 2026-08-08 04:07:36 on dist-test-slave-1l3l"
server_key: "f62e7a163e755414e6350ee9b8ff99f1"
server_key_iv: "2230815dfdcb2d9d111e02abc6595974"
server_key_version: "encryptionkey@0"
I20260808 04:07:36.020591 28146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/master-0/wal/instance:
uuid: "ce0c76f0336e4437aa940b8e18eedbab"
format_stamp: "Formatted at 2026-08-08 04:07:36 on dist-test-slave-1l3l"
server_key: "f62e7a163e755414e6350ee9b8ff99f1"
server_key_iv: "2230815dfdcb2d9d111e02abc6595974"
server_key_version: "encryptionkey@0"
I20260808 04:07:36.025432 28146 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 04:07:36.028693 28167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:07:36.029922 28146 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:07:36.030057 28146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/master-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/master-0/wal
uuid: "ce0c76f0336e4437aa940b8e18eedbab"
format_stamp: "Formatted at 2026-08-08 04:07:36 on dist-test-slave-1l3l"
server_key: "f62e7a163e755414e6350ee9b8ff99f1"
server_key_iv: "2230815dfdcb2d9d111e02abc6595974"
server_key_version: "encryptionkey@0"
I20260808 04:07:36.030222 28146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:07:36.074543 28146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:07:36.078631 28146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:07:36.078861 28146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:07:36.095821 28146 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.62:38749
I20260808 04:07:36.095840 28250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.62:38749 every 8 connection(s)
I20260808 04:07:36.097112 28146 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
I20260808 04:07:36.098719 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 28146
I20260808 04:07:36.098919 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/master-0/wal/instance
I20260808 04:07:36.099601 5452 external_mini_cluster.cc:1428] Setting key dc04503c145f7e3ecc1f24c392d5b3db
I20260808 04:07:36.101099 28252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:07:36.113772 28252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce0c76f0336e4437aa940b8e18eedbab: Bootstrap starting.
Aug 08 04:07:36 dist-test-slave-1l3l krb5kdc[28123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162055, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:07:36.117429 28252 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce0c76f0336e4437aa940b8e18eedbab: Neither blocks nor log segments found. Creating new log.
I20260808 04:07:36.118700 28252 log.cc:824] T 00000000000000000000000000000000 P ce0c76f0336e4437aa940b8e18eedbab: Log is configured to *not* fsync() on all Append() calls
I20260808 04:07:36.121846 28252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce0c76f0336e4437aa940b8e18eedbab: No bootstrap required, opened a new log
I20260808 04:07:36.122900 28255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:36.101508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42902 (local address 127.5.83.62:38749)
0808 04:07:36.102327 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0808 04:07:36.102342 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 04:07:36.102391 (+ 49us) server_negotiation.cc:407] Connection header received
0808 04:07:36.104271 (+ 1880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:07:36.104297 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:07:36.104780 (+ 483us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:07:36.105196 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:07:36.106553 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:36.108550 (+ 1997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:36.109832 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:36.110830 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:36.111009 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:36.114814 (+ 3805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:07:36.114825 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:07:36.114841 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:07:36.114893 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:07:36.118664 (+ 3771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:36.119355 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:36.119365 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:36.119371 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:36.119531 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:36.119985 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:36.119990 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:36.119993 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:36.120636 (+ 643us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:07:36.120908 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0808 04:07:36.121249 (+ 341us) server_negotiation.cc:299] Negotiation successful
0808 04:07:36.121647 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":354,"threads_started":1}
I20260808 04:07:36.130231 28252 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce0c76f0336e4437aa940b8e18eedbab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce0c76f0336e4437aa940b8e18eedbab" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 38749 } }
I20260808 04:07:36.130545 28252 raft_consensus.cc:374] T 00000000000000000000000000000000 P ce0c76f0336e4437aa940b8e18eedbab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:07:36.130622 28252 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce0c76f0336e4437aa940b8e18eedbab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce0c76f0336e4437aa940b8e18eedbab, State: Initialized, Role: FOLLOWER
I20260808 04:07:36.131115 28252 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce0c76f0336e4437aa940b8e18eedbab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce0c76f0336e4437aa940b8e18eedbab" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 38749 } }
I20260808 04:07:36.131315 28252 raft_consensus.cc:388] T 00000000000000000000000000000000 P ce0c76f0336e4437aa940b8e18eedbab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:07:36.131408 28252 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce0c76f0336e4437aa940b8e18eedbab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:07:36.131536 28252 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce0c76f0336e4437aa940b8e18eedbab [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:07:36.133685 28252 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce0c76f0336e4437aa940b8e18eedbab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce0c76f0336e4437aa940b8e18eedbab" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 38749 } }
I20260808 04:07:36.134117 28252 leader_election.cc:302] T 00000000000000000000000000000000 P ce0c76f0336e4437aa940b8e18eedbab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce0c76f0336e4437aa940b8e18eedbab; no voters:
I20260808 04:07:36.134711 28252 leader_election.cc:288] T 00000000000000000000000000000000 P ce0c76f0336e4437aa940b8e18eedbab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:07:36.134841 28258 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce0c76f0336e4437aa940b8e18eedbab [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:07:36.135648 28258 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce0c76f0336e4437aa940b8e18eedbab [term 1 LEADER]: Becoming Leader. State: Replica: ce0c76f0336e4437aa940b8e18eedbab, State: Running, Role: LEADER
I20260808 04:07:36.136158 28258 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce0c76f0336e4437aa940b8e18eedbab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce0c76f0336e4437aa940b8e18eedbab" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 38749 } }
I20260808 04:07:36.136672 28252 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce0c76f0336e4437aa940b8e18eedbab [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:07:36.138645 28262 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce0c76f0336e4437aa940b8e18eedbab [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce0c76f0336e4437aa940b8e18eedbab. Latest consensus state: current_term: 1 leader_uuid: "ce0c76f0336e4437aa940b8e18eedbab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce0c76f0336e4437aa940b8e18eedbab" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 38749 } } }
I20260808 04:07:36.139006 28262 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce0c76f0336e4437aa940b8e18eedbab [sys.catalog]: This master's current role is: LEADER
I20260808 04:07:36.139376 28260 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce0c76f0336e4437aa940b8e18eedbab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ce0c76f0336e4437aa940b8e18eedbab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce0c76f0336e4437aa940b8e18eedbab" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 38749 } } }
I20260808 04:07:36.139660 28260 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce0c76f0336e4437aa940b8e18eedbab [sys.catalog]: This master's current role is: LEADER
I20260808 04:07:36.141122 28268 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:07:36.145923 28268 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:07:36.154435 28268 catalog_manager.cc:1347] Generated new cluster ID: 24183d2003cc437ca80298fbee1ead54
I20260808 04:07:36.154526 28268 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:07:36.173138 28268 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:07:36.174335 28268 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:07:36.180022 28268 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ce0c76f0336e4437aa940b8e18eedbab: Generated new TSK 0
I20260808 04:07:36.180724 28268 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.83.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.1.keytab.
Entry for principal kudu/127.5.83.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.1.keytab.
I20260808 04:07:36.262578 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.1:0
--local_ip_for_outbound_sockets=127.5.83.1
--webserver_interface=127.5.83.1
--webserver_port=0
--tserver_master_addrs=127.5.83.62:38749
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.1.keytab
--principal=kudu/127.5.83.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:36095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:07:36.589689 28288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:07:36.589963 28288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:07:36.590060 28288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:07:36.597355 28288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:07:36.597466 28288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:07:36.597589 28288 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.1
I20260808 04:07:36.607035 28288 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:36095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.1.keytab
--principal=kudu/127.5.83.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.83.1
--webserver_port=0
--tserver_master_addrs=127.5.83.62:38749
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.1
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:07:36.608196 28288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:07:36.609380 28288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:07:36.621841 28295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:07:36.622265 28300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:07:36.622323 28296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:07:36.622622 28288 server_base.cc:1034] running on GCE node
I20260808 04:07:36.623208 28288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:07:36.624145 28288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:07:36.625370 28288 hybrid_clock.cc:648] HybridClock initialized: now 1786162056625314 us; error 44 us; skew 500 ppm
I20260808 04:07:36.625680 28288 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:07:36 dist-test-slave-1l3l krb5kdc[28123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162056, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:07:36.628733 28288 init.cc:377] Logged in from keytab as kudu/127.5.83.1@KRBTEST.COM (short username kudu)
I20260808 04:07:36.630117 28288 webserver.cc:460] Webserver started at http://127.5.83.1:46157/ using document root <none> and password file <none>
I20260808 04:07:36.630764 28288 fs_manager.cc:359] Metadata directory not provided
I20260808 04:07:36.630887 28288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:07:36.631197 28288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:07:36.634166 28288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-0/data/instance:
uuid: "b4fff30c1f1d4b2ab709870b18a9f6e2"
format_stamp: "Formatted at 2026-08-08 04:07:36 on dist-test-slave-1l3l"
server_key: "1305c3b5cdd664233a6c287c4b3d1a5c"
server_key_iv: "700083247ef2d76dd3a98c329e121f88"
server_key_version: "encryptionkey@0"
I20260808 04:07:36.634918 28288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-0/wal/instance:
uuid: "b4fff30c1f1d4b2ab709870b18a9f6e2"
format_stamp: "Formatted at 2026-08-08 04:07:36 on dist-test-slave-1l3l"
server_key: "1305c3b5cdd664233a6c287c4b3d1a5c"
server_key_iv: "700083247ef2d76dd3a98c329e121f88"
server_key_version: "encryptionkey@0"
I20260808 04:07:36.639935 28288 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260808 04:07:36.643486 28311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:07:36.644747 28288 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:07:36.644898 28288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-0/wal
uuid: "b4fff30c1f1d4b2ab709870b18a9f6e2"
format_stamp: "Formatted at 2026-08-08 04:07:36 on dist-test-slave-1l3l"
server_key: "1305c3b5cdd664233a6c287c4b3d1a5c"
server_key_iv: "700083247ef2d76dd3a98c329e121f88"
server_key_version: "encryptionkey@0"
I20260808 04:07:36.645074 28288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:07:36.663740 28288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:07:36.667304 28288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:07:36.667577 28288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:07:36.668510 28288 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:07:36.669783 28288 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:07:36.669857 28288 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:07:36.669982 28288 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:07:36.670039 28288 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:07:36.701283 28288 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.1:34443
I20260808 04:07:36.701339 28474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.1:34443 every 8 connection(s)
I20260808 04:07:36.702492 28288 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
I20260808 04:07:36.709388 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 28288
I20260808 04:07:36.709597 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-0/wal/instance
I20260808 04:07:36.710247 5452 external_mini_cluster.cc:1428] Setting key 392fe99fe7fc4e091046025661173076
Aug 08 04:07:36 dist-test-slave-1l3l krb5kdc[28123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162056, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.1@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:07:36.721604 28478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:36.704872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.1:54353 (local address 127.5.83.62:38749)
0808 04:07:36.705287 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0808 04:07:36.705292 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:07:36.706218 (+ 926us) server_negotiation.cc:407] Connection header received
0808 04:07:36.708023 (+ 1805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:07:36.708029 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:07:36.708106 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:07:36.708202 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:07:36.709637 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:36.710359 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:36.712054 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:36.712454 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:36.712560 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:36.716131 (+ 3571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:07:36.716137 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:07:36.716140 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:07:36.716172 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:07:36.718701 (+ 2529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:36.719483 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:36.719487 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:36.719489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:36.719621 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:36.720074 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:36.720077 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:36.720079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:36.720334 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:07:36.720443 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0808 04:07:36.721152 (+ 709us) server_negotiation.cc:299] Negotiation successful
0808 04:07:36.721352 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":215,"threads_started":1}
I20260808 04:07:36.722748 28479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:36.705294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:38749 (local address 127.5.83.1:54353)
0808 04:07:36.706059 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:36.706101 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 04:07:36.707755 (+ 1654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:07:36.708427 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:07:36.708435 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:07:36.708929 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:07:36.709457 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:07:36.709479 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:07:36.710504 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:07:36.710508 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:07:36.711928 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:07:36.711940 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:07:36.712667 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:07:36.712674 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:07:36.712889 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:36.713498 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:07:36.713524 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:07:36.715896 (+ 2372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:07:36.718833 (+ 2937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:07:36.718841 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:07:36.718855 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:07:36.719352 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:07:36.719739 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:07:36.719743 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:07:36.719746 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:07:36.719976 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:07:36.720451 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:07:36.720458 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:07:36.720822 (+ 364us) client_negotiation.cc:769] Sending connection context
0808 04:07:36.721086 (+ 264us) client_negotiation.cc:240] Negotiation successful
0808 04:07:36.721552 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":275,"threads_started":1}
I20260808 04:07:36.724164 28476 heartbeater.cc:347] Connected to a master server at 127.5.83.62:38749
I20260808 04:07:36.724436 28476 heartbeater.cc:464] Registering TS with master...
I20260808 04:07:36.725121 28476 heartbeater.cc:511] Master 127.5.83.62:38749 requested a full tablet report, sending...
I20260808 04:07:36.727118 28193 ts_manager.cc:194] Registered new tserver with Master: b4fff30c1f1d4b2ab709870b18a9f6e2 (127.5.83.1:34443)
I20260808 04:07:36.728544 28193 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.1@KRBTEST.COM'} at 127.5.83.1:54353
WARNING: no policy specified for kudu/127.5.83.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.2.keytab.
Entry for principal kudu/127.5.83.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.2.keytab.
I20260808 04:07:36.766786 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.2:0
--local_ip_for_outbound_sockets=127.5.83.2
--webserver_interface=127.5.83.2
--webserver_port=0
--tserver_master_addrs=127.5.83.62:38749
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.2.keytab
--principal=kudu/127.5.83.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:36095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:07:37.092763 28483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:07:37.093039 28483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:07:37.093179 28483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:07:37.100137 28483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:07:37.100244 28483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:07:37.100409 28483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.2
I20260808 04:07:37.109651 28483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:36095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.2.keytab
--principal=kudu/127.5.83.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.83.2
--webserver_port=0
--tserver_master_addrs=127.5.83.62:38749
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.2
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:07:37.110769 28483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:07:37.112036 28483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:07:37.124465 28488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:07:37.124433 28490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:07:37.124500 28493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:07:37.125279 28483 server_base.cc:1034] running on GCE node
I20260808 04:07:37.125813 28483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:07:37.126669 28483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:07:37.127866 28483 hybrid_clock.cc:648] HybridClock initialized: now 1786162057127823 us; error 28 us; skew 500 ppm
I20260808 04:07:37.128172 28483 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:07:37 dist-test-slave-1l3l krb5kdc[28123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162057, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:07:37.131142 28483 init.cc:377] Logged in from keytab as kudu/127.5.83.2@KRBTEST.COM (short username kudu)
I20260808 04:07:37.132591 28483 webserver.cc:460] Webserver started at http://127.5.83.2:39145/ using document root <none> and password file <none>
I20260808 04:07:37.133212 28483 fs_manager.cc:359] Metadata directory not provided
I20260808 04:07:37.133328 28483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:07:37.133651 28483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:07:37.136515 28483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-1/data/instance:
uuid: "f3a563398cf94de0ae7d1c748d99faab"
format_stamp: "Formatted at 2026-08-08 04:07:37 on dist-test-slave-1l3l"
server_key: "4509fe7a497d918247ab6a2645d7c1ec"
server_key_iv: "3ae1d35ecc78a2fee8661d6c1b9613d9"
server_key_version: "encryptionkey@0"
I20260808 04:07:37.137204 28483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-1/wal/instance:
uuid: "f3a563398cf94de0ae7d1c748d99faab"
format_stamp: "Formatted at 2026-08-08 04:07:37 on dist-test-slave-1l3l"
server_key: "4509fe7a497d918247ab6a2645d7c1ec"
server_key_iv: "3ae1d35ecc78a2fee8661d6c1b9613d9"
server_key_version: "encryptionkey@0"
I20260808 04:07:37.142171 28483 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 04:07:37.145371 28502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:07:37.146605 28483 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 04:07:37.146777 28483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-1/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-1/wal
uuid: "f3a563398cf94de0ae7d1c748d99faab"
format_stamp: "Formatted at 2026-08-08 04:07:37 on dist-test-slave-1l3l"
server_key: "4509fe7a497d918247ab6a2645d7c1ec"
server_key_iv: "3ae1d35ecc78a2fee8661d6c1b9613d9"
server_key_version: "encryptionkey@0"
I20260808 04:07:37.146991 28483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:07:37.162384 28483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:07:37.165932 28483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:07:37.166160 28483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:07:37.166981 28483 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:07:37.168213 28483 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:07:37.168284 28483 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:07:37.168387 28483 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:07:37.168439 28483 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:07:37.198064 28483 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.2:33807
I20260808 04:07:37.198112 28678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.2:33807 every 8 connection(s)
I20260808 04:07:37.199281 28483 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
I20260808 04:07:37.200403 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 28483
I20260808 04:07:37.200621 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-1/wal/instance
I20260808 04:07:37.201241 5452 external_mini_cluster.cc:1428] Setting key 6f23d4506357bba86d81400c6ffdebc6
Aug 08 04:07:37 dist-test-slave-1l3l krb5kdc[28123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162057, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.2@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:07:37.219367 28478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:37.202217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.2:59105 (local address 127.5.83.62:38749)
0808 04:07:37.202424 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0808 04:07:37.202428 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:07:37.203941 (+ 1513us) server_negotiation.cc:407] Connection header received
0808 04:07:37.205760 (+ 1819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:07:37.205766 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:07:37.205845 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:07:37.205945 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:07:37.207446 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:37.208418 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:37.209922 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:37.210526 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:37.210644 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:37.213839 (+ 3195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:07:37.213845 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:07:37.213849 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:07:37.213885 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:07:37.216507 (+ 2622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:37.217203 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:37.217209 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:37.217211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:37.217340 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:37.217761 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:37.217766 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:37.217768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:37.218026 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:07:37.218161 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0808 04:07:37.218790 (+ 629us) server_negotiation.cc:299] Negotiation successful
0808 04:07:37.219020 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260808 04:07:37.220279 28681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:37.202817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:38749 (local address 127.5.83.2:59105)
0808 04:07:37.203785 (+ 968us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:37.203834 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0808 04:07:37.205505 (+ 1671us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:07:37.206174 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:07:37.206182 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:07:37.206736 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:07:37.207262 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:07:37.207280 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:07:37.208576 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:07:37.208580 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:07:37.209778 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:07:37.209790 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:07:37.210702 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:07:37.210707 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:07:37.210912 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:37.211526 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:07:37.211548 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:07:37.213628 (+ 2080us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:07:37.216637 (+ 3009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:07:37.216643 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:07:37.216653 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:07:37.217086 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:07:37.217467 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:07:37.217472 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:07:37.217476 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:07:37.217655 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:07:37.218149 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:07:37.218156 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:07:37.218452 (+ 296us) client_negotiation.cc:769] Sending connection context
0808 04:07:37.218694 (+ 242us) client_negotiation.cc:240] Negotiation successful
0808 04:07:37.219168 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":681,"thread_start_us":438,"threads_started":1}
I20260808 04:07:37.221582 28679 heartbeater.cc:347] Connected to a master server at 127.5.83.62:38749
I20260808 04:07:37.221836 28679 heartbeater.cc:464] Registering TS with master...
I20260808 04:07:37.222506 28679 heartbeater.cc:511] Master 127.5.83.62:38749 requested a full tablet report, sending...
I20260808 04:07:37.223889 28193 ts_manager.cc:194] Registered new tserver with Master: f3a563398cf94de0ae7d1c748d99faab (127.5.83.2:33807)
I20260808 04:07:37.224676 28193 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.2@KRBTEST.COM'} at 127.5.83.2:59105
WARNING: no policy specified for kudu/127.5.83.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.3.keytab.
Entry for principal kudu/127.5.83.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.3.keytab.
I20260808 04:07:37.260062 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.3:0
--local_ip_for_outbound_sockets=127.5.83.3
--webserver_interface=127.5.83.3
--webserver_port=0
--tserver_master_addrs=127.5.83.62:38749
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.3.keytab
--principal=kudu/127.5.83.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:36095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:07:37.590981 28684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:07:37.591255 28684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:07:37.591358 28684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:07:37.598543 28684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:07:37.598644 28684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:07:37.598762 28684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.3
I20260808 04:07:37.608268 28684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:36095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.3.keytab
--principal=kudu/127.5.83.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.83.3
--webserver_port=0
--tserver_master_addrs=127.5.83.62:38749
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.3
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:07:37.609454 28684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:07:37.610666 28684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:07:37.622974 28691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:07:37.623020 28693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:07:37.623018 28698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:07:37.624117 28684 server_base.cc:1034] running on GCE node
I20260808 04:07:37.624714 28684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:07:37.625582 28684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:07:37.626796 28684 hybrid_clock.cc:648] HybridClock initialized: now 1786162057626738 us; error 49 us; skew 500 ppm
I20260808 04:07:37.627104 28684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:07:37 dist-test-slave-1l3l krb5kdc[28123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162057, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:07:37.630121 28684 init.cc:377] Logged in from keytab as kudu/127.5.83.3@KRBTEST.COM (short username kudu)
I20260808 04:07:37.631626 28684 webserver.cc:460] Webserver started at http://127.5.83.3:34739/ using document root <none> and password file <none>
I20260808 04:07:37.632301 28684 fs_manager.cc:359] Metadata directory not provided
I20260808 04:07:37.632437 28684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:07:37.632740 28684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:07:37.635686 28684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-2/data/instance:
uuid: "87132debb51140cb9d97a17d217b6852"
format_stamp: "Formatted at 2026-08-08 04:07:37 on dist-test-slave-1l3l"
server_key: "5414e96a2625ef619cf6815a5709b6b9"
server_key_iv: "b801ea8a49fd8e0ce02a7c5f1f2dbb1c"
server_key_version: "encryptionkey@0"
I20260808 04:07:37.636550 28684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-2/wal/instance:
uuid: "87132debb51140cb9d97a17d217b6852"
format_stamp: "Formatted at 2026-08-08 04:07:37 on dist-test-slave-1l3l"
server_key: "5414e96a2625ef619cf6815a5709b6b9"
server_key_iv: "b801ea8a49fd8e0ce02a7c5f1f2dbb1c"
server_key_version: "encryptionkey@0"
I20260808 04:07:37.641603 28684 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 04:07:37.644877 28708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:07:37.646075 28684 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260808 04:07:37.646229 28684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-2/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-2/wal
uuid: "87132debb51140cb9d97a17d217b6852"
format_stamp: "Formatted at 2026-08-08 04:07:37 on dist-test-slave-1l3l"
server_key: "5414e96a2625ef619cf6815a5709b6b9"
server_key_iv: "b801ea8a49fd8e0ce02a7c5f1f2dbb1c"
server_key_version: "encryptionkey@0"
I20260808 04:07:37.646427 28684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:07:37.662990 28684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:07:37.666733 28684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:07:37.666993 28684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:07:37.667958 28684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:07:37.669236 28684 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:07:37.669317 28684 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:07:37.669445 28684 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:07:37.669514 28684 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:07:37.699534 28684 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.3:42143
I20260808 04:07:37.699558 28888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.3:42143 every 8 connection(s)
I20260808 04:07:37.700762 28684 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
I20260808 04:07:37.703859 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 28684
I20260808 04:07:37.704035 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-2/wal/instance
I20260808 04:07:37.704627 5452 external_mini_cluster.cc:1428] Setting key 7e3ec3400c0fc54bb6dcab707d239c93
Aug 08 04:07:37 dist-test-slave-1l3l krb5kdc[28123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162057, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.3@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:07:37.720166 28478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:37.703339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.3:46997 (local address 127.5.83.62:38749)
0808 04:07:37.703528 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0808 04:07:37.703534 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:07:37.705052 (+ 1518us) server_negotiation.cc:407] Connection header received
0808 04:07:37.707157 (+ 2105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:07:37.707163 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:07:37.707243 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:07:37.707350 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:07:37.708810 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:37.709587 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:37.711037 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:37.711407 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:37.711496 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:37.714665 (+ 3169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:07:37.714670 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:07:37.714673 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:07:37.714708 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:07:37.717112 (+ 2404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:37.717740 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:37.717744 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:37.717746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:37.717864 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:37.718342 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:37.718346 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:37.718349 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:37.718612 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:07:37.718743 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0808 04:07:37.719615 (+ 872us) server_negotiation.cc:299] Negotiation successful
0808 04:07:37.719842 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 04:07:37.720958 28892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:37.703901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:38749 (local address 127.5.83.3:46997)
0808 04:07:37.704846 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:37.704919 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0808 04:07:37.706914 (+ 1995us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:07:37.707564 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:07:37.707572 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:07:37.708139 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:07:37.708658 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:07:37.708674 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:07:37.709726 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:07:37.709730 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:07:37.710928 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:07:37.710940 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:07:37.711570 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:07:37.711574 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:07:37.711801 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:37.712381 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:07:37.712403 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:07:37.714480 (+ 2077us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:07:37.717231 (+ 2751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:07:37.717237 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:07:37.717247 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:07:37.717648 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:07:37.717974 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:07:37.717979 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:07:37.717982 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:07:37.718222 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:07:37.718736 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:07:37.718743 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:07:37.719131 (+ 388us) client_negotiation.cc:769] Sending connection context
0808 04:07:37.719439 (+ 308us) client_negotiation.cc:240] Negotiation successful
0808 04:07:37.719940 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":348,"threads_started":1}
I20260808 04:07:37.722151 28889 heartbeater.cc:347] Connected to a master server at 127.5.83.62:38749
I20260808 04:07:37.722445 28889 heartbeater.cc:464] Registering TS with master...
I20260808 04:07:37.723124 28889 heartbeater.cc:511] Master 127.5.83.62:38749 requested a full tablet report, sending...
I20260808 04:07:37.724478 28193 ts_manager.cc:194] Registered new tserver with Master: 87132debb51140cb9d97a17d217b6852 (127.5.83.3:42143)
I20260808 04:07:37.725253 28193 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.3@KRBTEST.COM'} at 127.5.83.3:46997
I20260808 04:07:37.731139 28476 heartbeater.cc:503] Master 127.5.83.62:38749 was elected leader, sending a full tablet report...
I20260808 04:07:37.732318 5452 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:07:37.748737 28478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:37.736763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42916 (local address 127.5.83.62:38749)
0808 04:07:37.736927 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0808 04:07:37.736932 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:07:37.737424 (+ 492us) server_negotiation.cc:407] Connection header received
0808 04:07:37.737543 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:07:37.737547 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:07:37.737602 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:07:37.737676 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:07:37.739111 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:37.739862 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:37.741326 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:37.741723 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:37.741820 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:37.744080 (+ 2260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:07:37.744085 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:07:37.744088 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:07:37.744116 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:07:37.746393 (+ 2277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:37.747088 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:37.747091 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:37.747093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:37.747197 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:37.747475 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:37.747478 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:37.747480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:37.747798 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:07:37.747965 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0808 04:07:37.748269 (+ 304us) server_negotiation.cc:299] Negotiation successful
0808 04:07:37.748435 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 04:07:37.752775 28191 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42916:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:07:37.759474 28191 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:07:37.798607 28906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:37.770767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:34443 (local address 127.0.0.1:52148)
0808 04:07:37.788183 (+ 17416us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:37.788204 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 04:07:37.788329 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:07:37.791268 (+ 2939us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:07:37.791277 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:07:37.791313 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:07:37.792073 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:07:37.792090 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:07:37.794052 (+ 1962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:07:37.794056 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:07:37.796272 (+ 2216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:07:37.796288 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:07:37.797686 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:07:37.797693 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:07:37.797906 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:37.797969 (+ 63us) client_negotiation.cc:769] Sending connection context
0808 04:07:37.798085 (+ 116us) client_negotiation.cc:240] Negotiation successful
0808 04:07:37.798253 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17285,"thread_start_us":17117,"threads_started":1}
I20260808 04:07:37.798926 28909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:37.771299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52148 (local address 127.5.83.1:34443)
0808 04:07:37.790282 (+ 18983us) server_negotiation.cc:206] Beginning negotiation
0808 04:07:37.790292 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 04:07:37.790336 (+ 44us) server_negotiation.cc:407] Connection header received
0808 04:07:37.790455 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:07:37.790464 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:07:37.790723 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:07:37.790938 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:07:37.792522 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:37.793923 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:37.796430 (+ 2507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:37.797479 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:37.797605 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:37.797835 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0808 04:07:37.798172 (+ 337us) server_negotiation.cc:299] Negotiation successful
0808 04:07:37.798347 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18783,"thread_start_us":20935,"threads_started":1}
I20260808 04:07:37.799927 28912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:37.771833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41510 (local address 127.5.83.2:33807)
0808 04:07:37.780702 (+ 8869us) server_negotiation.cc:206] Beginning negotiation
0808 04:07:37.780713 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 04:07:37.792311 (+ 11598us) server_negotiation.cc:407] Connection header received
0808 04:07:37.792475 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:07:37.792483 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:07:37.792697 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:07:37.792919 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:07:37.793505 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:37.794739 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:37.797241 (+ 2502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:37.798572 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:37.798693 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:37.798869 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0808 04:07:37.799410 (+ 541us) server_negotiation.cc:299] Negotiation successful
0808 04:07:37.799570 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8679,"thread_start_us":8529,"threads_started":1}
I20260808 04:07:37.800688 28911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:37.771236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.2:33807 (local address 127.0.0.1:41510)
0808 04:07:37.792187 (+ 20951us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:37.792201 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:07:37.792326 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:07:37.792916 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:07:37.792921 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:07:37.792946 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:07:37.793318 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:07:37.793330 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:07:37.794915 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:07:37.794919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:07:37.797067 (+ 2148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:07:37.797083 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:07:37.799031 (+ 1948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:07:37.799038 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:07:37.799193 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:37.799230 (+ 37us) client_negotiation.cc:769] Sending connection context
0808 04:07:37.800192 (+ 962us) client_negotiation.cc:240] Negotiation successful
0808 04:07:37.800320 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20827,"mutex_wait_us":19,"thread_start_us":20665,"threads_started":1}
I20260808 04:07:37.800741 28905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:37.770626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.3:42143 (local address 127.0.0.1:53496)
0808 04:07:37.782847 (+ 12221us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:37.782891 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 04:07:37.783095 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:07:37.793625 (+ 10530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:07:37.793629 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:07:37.793651 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:07:37.793942 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:07:37.793951 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:07:37.795852 (+ 1901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:07:37.795858 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:07:37.797741 (+ 1883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:07:37.797753 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:07:37.799875 (+ 2122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:07:37.799882 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:07:37.800044 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:37.800076 (+ 32us) client_negotiation.cc:769] Sending connection context
0808 04:07:37.800159 (+ 83us) client_negotiation.cc:240] Negotiation successful
0808 04:07:37.800296 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11956,"mutex_wait_us":25,"thread_start_us":12644,"threads_started":1}
I20260808 04:07:37.802099 28908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:37.770853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53496 (local address 127.5.83.3:42143)
0808 04:07:37.793097 (+ 22244us) server_negotiation.cc:206] Beginning negotiation
0808 04:07:37.793106 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 04:07:37.793135 (+ 29us) server_negotiation.cc:407] Connection header received
0808 04:07:37.793227 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:07:37.793233 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:07:37.793454 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:07:37.793622 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:07:37.794118 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:37.795693 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:37.797920 (+ 2227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:37.799659 (+ 1739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:37.799806 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:37.801696 (+ 1890us) server_negotiation.cc:1035] Waiting for connection context
0808 04:07:37.801753 (+ 57us) server_negotiation.cc:299] Negotiation successful
0808 04:07:37.801858 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22073,"thread_start_us":21762,"threads_started":1}
I20260808 04:07:37.802819 28381 tablet_service.cc:1467] Processing CreateTablet for tablet 545b8ba3bbe24ec988ba8664704ffee7 (DEFAULT_TABLE table=test-table [id=7892f64c9b8d4945945d765f566e46a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:07:37.804648 28381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 545b8ba3bbe24ec988ba8664704ffee7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:07:37.804224 28573 tablet_service.cc:1467] Processing CreateTablet for tablet 545b8ba3bbe24ec988ba8664704ffee7 (DEFAULT_TABLE table=test-table [id=7892f64c9b8d4945945d765f566e46a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:07:37.805063 28778 tablet_service.cc:1467] Processing CreateTablet for tablet 545b8ba3bbe24ec988ba8664704ffee7 (DEFAULT_TABLE table=test-table [id=7892f64c9b8d4945945d765f566e46a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:07:37.805555 28573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 545b8ba3bbe24ec988ba8664704ffee7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:07:37.806151 28778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 545b8ba3bbe24ec988ba8664704ffee7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:07:37.816126 28913 tablet_bootstrap.cc:492] T 545b8ba3bbe24ec988ba8664704ffee7 P 87132debb51140cb9d97a17d217b6852: Bootstrap starting.
I20260808 04:07:37.819136 28914 tablet_bootstrap.cc:492] T 545b8ba3bbe24ec988ba8664704ffee7 P f3a563398cf94de0ae7d1c748d99faab: Bootstrap starting.
I20260808 04:07:37.820461 28913 tablet_bootstrap.cc:654] T 545b8ba3bbe24ec988ba8664704ffee7 P 87132debb51140cb9d97a17d217b6852: Neither blocks nor log segments found. Creating new log.
I20260808 04:07:37.821681 28913 log.cc:824] T 545b8ba3bbe24ec988ba8664704ffee7 P 87132debb51140cb9d97a17d217b6852: Log is configured to *not* fsync() on all Append() calls
I20260808 04:07:37.822535 28914 tablet_bootstrap.cc:654] T 545b8ba3bbe24ec988ba8664704ffee7 P f3a563398cf94de0ae7d1c748d99faab: Neither blocks nor log segments found. Creating new log.
I20260808 04:07:37.823729 28914 log.cc:824] T 545b8ba3bbe24ec988ba8664704ffee7 P f3a563398cf94de0ae7d1c748d99faab: Log is configured to *not* fsync() on all Append() calls
I20260808 04:07:37.823047 28915 tablet_bootstrap.cc:492] T 545b8ba3bbe24ec988ba8664704ffee7 P b4fff30c1f1d4b2ab709870b18a9f6e2: Bootstrap starting.
I20260808 04:07:37.824792 28913 tablet_bootstrap.cc:492] T 545b8ba3bbe24ec988ba8664704ffee7 P 87132debb51140cb9d97a17d217b6852: No bootstrap required, opened a new log
I20260808 04:07:37.825361 28913 ts_tablet_manager.cc:1397] T 545b8ba3bbe24ec988ba8664704ffee7 P 87132debb51140cb9d97a17d217b6852: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260808 04:07:37.826716 28915 tablet_bootstrap.cc:654] T 545b8ba3bbe24ec988ba8664704ffee7 P b4fff30c1f1d4b2ab709870b18a9f6e2: Neither blocks nor log segments found. Creating new log.
I20260808 04:07:37.826804 28914 tablet_bootstrap.cc:492] T 545b8ba3bbe24ec988ba8664704ffee7 P f3a563398cf94de0ae7d1c748d99faab: No bootstrap required, opened a new log
I20260808 04:07:37.827059 28914 ts_tablet_manager.cc:1397] T 545b8ba3bbe24ec988ba8664704ffee7 P f3a563398cf94de0ae7d1c748d99faab: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260808 04:07:37.827668 28915 log.cc:824] T 545b8ba3bbe24ec988ba8664704ffee7 P b4fff30c1f1d4b2ab709870b18a9f6e2: Log is configured to *not* fsync() on all Append() calls
I20260808 04:07:37.829908 28915 tablet_bootstrap.cc:492] T 545b8ba3bbe24ec988ba8664704ffee7 P b4fff30c1f1d4b2ab709870b18a9f6e2: No bootstrap required, opened a new log
I20260808 04:07:37.830127 28915 ts_tablet_manager.cc:1397] T 545b8ba3bbe24ec988ba8664704ffee7 P b4fff30c1f1d4b2ab709870b18a9f6e2: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260808 04:07:37.835615 28913 raft_consensus.cc:348] T 545b8ba3bbe24ec988ba8664704ffee7 P 87132debb51140cb9d97a17d217b6852 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87132debb51140cb9d97a17d217b6852" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 42143 } } peers { permanent_uuid: "b4fff30c1f1d4b2ab709870b18a9f6e2" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 34443 } } peers { permanent_uuid: "f3a563398cf94de0ae7d1c748d99faab" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 33807 } }
I20260808 04:07:37.836059 28913 raft_consensus.cc:374] T 545b8ba3bbe24ec988ba8664704ffee7 P 87132debb51140cb9d97a17d217b6852 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:07:37.836149 28913 raft_consensus.cc:729] T 545b8ba3bbe24ec988ba8664704ffee7 P 87132debb51140cb9d97a17d217b6852 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87132debb51140cb9d97a17d217b6852, State: Initialized, Role: FOLLOWER
I20260808 04:07:37.836864 28913 consensus_queue.cc:260] T 545b8ba3bbe24ec988ba8664704ffee7 P 87132debb51140cb9d97a17d217b6852 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87132debb51140cb9d97a17d217b6852" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 42143 } } peers { permanent_uuid: "b4fff30c1f1d4b2ab709870b18a9f6e2" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 34443 } } peers { permanent_uuid: "f3a563398cf94de0ae7d1c748d99faab" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 33807 } }
I20260808 04:07:37.837133 28914 raft_consensus.cc:348] T 545b8ba3bbe24ec988ba8664704ffee7 P f3a563398cf94de0ae7d1c748d99faab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87132debb51140cb9d97a17d217b6852" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 42143 } } peers { permanent_uuid: "b4fff30c1f1d4b2ab709870b18a9f6e2" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 34443 } } peers { permanent_uuid: "f3a563398cf94de0ae7d1c748d99faab" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 33807 } }
I20260808 04:07:37.837625 28914 raft_consensus.cc:374] T 545b8ba3bbe24ec988ba8664704ffee7 P f3a563398cf94de0ae7d1c748d99faab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:07:37.837733 28914 raft_consensus.cc:729] T 545b8ba3bbe24ec988ba8664704ffee7 P f3a563398cf94de0ae7d1c748d99faab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3a563398cf94de0ae7d1c748d99faab, State: Initialized, Role: FOLLOWER
I20260808 04:07:37.837584 28915 raft_consensus.cc:348] T 545b8ba3bbe24ec988ba8664704ffee7 P b4fff30c1f1d4b2ab709870b18a9f6e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87132debb51140cb9d97a17d217b6852" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 42143 } } peers { permanent_uuid: "b4fff30c1f1d4b2ab709870b18a9f6e2" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 34443 } } peers { permanent_uuid: "f3a563398cf94de0ae7d1c748d99faab" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 33807 } }
I20260808 04:07:37.838001 28915 raft_consensus.cc:374] T 545b8ba3bbe24ec988ba8664704ffee7 P b4fff30c1f1d4b2ab709870b18a9f6e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:07:37.838105 28915 raft_consensus.cc:729] T 545b8ba3bbe24ec988ba8664704ffee7 P b4fff30c1f1d4b2ab709870b18a9f6e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4fff30c1f1d4b2ab709870b18a9f6e2, State: Initialized, Role: FOLLOWER
I20260808 04:07:37.838281 28889 heartbeater.cc:503] Master 127.5.83.62:38749 was elected leader, sending a full tablet report...
I20260808 04:07:37.838661 28913 ts_tablet_manager.cc:1428] T 545b8ba3bbe24ec988ba8664704ffee7 P 87132debb51140cb9d97a17d217b6852: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20260808 04:07:37.838603 28915 consensus_queue.cc:260] T 545b8ba3bbe24ec988ba8664704ffee7 P b4fff30c1f1d4b2ab709870b18a9f6e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87132debb51140cb9d97a17d217b6852" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 42143 } } peers { permanent_uuid: "b4fff30c1f1d4b2ab709870b18a9f6e2" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 34443 } } peers { permanent_uuid: "f3a563398cf94de0ae7d1c748d99faab" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 33807 } }
I20260808 04:07:37.839212 28914 consensus_queue.cc:260] T 545b8ba3bbe24ec988ba8664704ffee7 P f3a563398cf94de0ae7d1c748d99faab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87132debb51140cb9d97a17d217b6852" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 42143 } } peers { permanent_uuid: "b4fff30c1f1d4b2ab709870b18a9f6e2" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 34443 } } peers { permanent_uuid: "f3a563398cf94de0ae7d1c748d99faab" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 33807 } }
I20260808 04:07:37.840159 28915 ts_tablet_manager.cc:1428] T 545b8ba3bbe24ec988ba8664704ffee7 P b4fff30c1f1d4b2ab709870b18a9f6e2: Time spent starting tablet: real 0.010s user 0.006s sys 0.005s
I20260808 04:07:37.846185 28679 heartbeater.cc:503] Master 127.5.83.62:38749 was elected leader, sending a full tablet report...
I20260808 04:07:37.846803 28914 ts_tablet_manager.cc:1428] T 545b8ba3bbe24ec988ba8664704ffee7 P f3a563398cf94de0ae7d1c748d99faab: Time spent starting tablet: real 0.020s user 0.011s sys 0.008s
I20260808 04:07:37.859078 28921 raft_consensus.cc:482] T 545b8ba3bbe24ec988ba8664704ffee7 P 87132debb51140cb9d97a17d217b6852 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:07:37.859292 28921 raft_consensus.cc:504] T 545b8ba3bbe24ec988ba8664704ffee7 P 87132debb51140cb9d97a17d217b6852 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87132debb51140cb9d97a17d217b6852" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 42143 } } peers { permanent_uuid: "b4fff30c1f1d4b2ab709870b18a9f6e2" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 34443 } } peers { permanent_uuid: "f3a563398cf94de0ae7d1c748d99faab" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 33807 } }
I20260808 04:07:37.860772 28921 leader_election.cc:288] T 545b8ba3bbe24ec988ba8664704ffee7 P 87132debb51140cb9d97a17d217b6852 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b4fff30c1f1d4b2ab709870b18a9f6e2 (127.5.83.1:34443), f3a563398cf94de0ae7d1c748d99faab (127.5.83.2:33807)
I20260808 04:07:37.866123 28892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:37.860964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.2:33807 (local address 127.5.83.3:52839)
0808 04:07:37.861160 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:37.861180 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 04:07:37.861274 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:07:37.861627 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:07:37.861632 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:07:37.861664 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:07:37.861894 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:07:37.861907 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:07:37.862864 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:07:37.862869 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:07:37.864452 (+ 1583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:07:37.864467 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:07:37.865613 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:07:37.865617 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:07:37.865743 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:37.865767 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 04:07:37.865845 (+ 78us) client_negotiation.cc:240] Negotiation successful
0808 04:07:37.865932 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260808 04:07:37.866262 28912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:37.861102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.3:52839 (local address 127.5.83.2:33807)
0808 04:07:37.861299 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0808 04:07:37.861305 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:07:37.861324 (+ 19us) server_negotiation.cc:407] Connection header received
0808 04:07:37.861386 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:07:37.861391 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:07:37.861493 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:07:37.861613 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:07:37.862036 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:37.862728 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:37.864611 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:37.865470 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:37.865601 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:37.865693 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0808 04:07:37.865882 (+ 189us) server_negotiation.cc:299] Negotiation successful
0808 04:07:37.866027 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 04:07:37.867151 28606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "545b8ba3bbe24ec988ba8664704ffee7" candidate_uuid: "87132debb51140cb9d97a17d217b6852" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3a563398cf94de0ae7d1c748d99faab" is_pre_election: true
I20260808 04:07:37.867556 28909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:37.861299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.3:46147 (local address 127.5.83.1:34443)
0808 04:07:37.861482 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0808 04:07:37.861486 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:07:37.862125 (+ 639us) server_negotiation.cc:407] Connection header received
0808 04:07:37.862276 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:07:37.862280 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:07:37.862335 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:07:37.862404 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:07:37.862875 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:37.863558 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:37.865583 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:37.866743 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:37.866850 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:37.866921 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0808 04:07:37.867258 (+ 337us) server_negotiation.cc:299] Negotiation successful
0808 04:07:37.867342 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 04:07:37.867738 28925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:37.860987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:34443 (local address 127.5.83.3:46147)
0808 04:07:37.862039 (+ 1052us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:37.862055 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:07:37.862158 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:07:37.862433 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:07:37.862437 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:07:37.862460 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:07:37.862729 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:07:37.862737 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:07:37.863726 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:07:37.863732 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:07:37.865434 (+ 1702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:07:37.865447 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:07:37.866949 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:07:37.866955 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:07:37.867112 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:37.867146 (+ 34us) client_negotiation.cc:769] Sending connection context
0808 04:07:37.867249 (+ 103us) client_negotiation.cc:240] Negotiation successful
0808 04:07:37.867368 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":931,"mutex_wait_us":383,"thread_start_us":298,"threads_started":1}
I20260808 04:07:37.867671 28606 raft_consensus.cc:2454] T 545b8ba3bbe24ec988ba8664704ffee7 P f3a563398cf94de0ae7d1c748d99faab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87132debb51140cb9d97a17d217b6852 in term 0.
I20260808 04:07:37.868312 28713 leader_election.cc:302] T 545b8ba3bbe24ec988ba8664704ffee7 P 87132debb51140cb9d97a17d217b6852 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87132debb51140cb9d97a17d217b6852, f3a563398cf94de0ae7d1c748d99faab; no voters:
I20260808 04:07:37.868649 28921 raft_consensus.cc:2781] T 545b8ba3bbe24ec988ba8664704ffee7 P 87132debb51140cb9d97a17d217b6852 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:07:37.868763 28921 raft_consensus.cc:482] T 545b8ba3bbe24ec988ba8664704ffee7 P 87132debb51140cb9d97a17d217b6852 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:07:37.868860 28921 raft_consensus.cc:3037] T 545b8ba3bbe24ec988ba8664704ffee7 P 87132debb51140cb9d97a17d217b6852 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:07:37.868697 28409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "545b8ba3bbe24ec988ba8664704ffee7" candidate_uuid: "87132debb51140cb9d97a17d217b6852" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4fff30c1f1d4b2ab709870b18a9f6e2" is_pre_election: true
I20260808 04:07:37.869138 28409 raft_consensus.cc:2454] T 545b8ba3bbe24ec988ba8664704ffee7 P b4fff30c1f1d4b2ab709870b18a9f6e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87132debb51140cb9d97a17d217b6852 in term 0.
I20260808 04:07:37.871201 28921 raft_consensus.cc:504] T 545b8ba3bbe24ec988ba8664704ffee7 P 87132debb51140cb9d97a17d217b6852 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87132debb51140cb9d97a17d217b6852" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 42143 } } peers { permanent_uuid: "b4fff30c1f1d4b2ab709870b18a9f6e2" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 34443 } } peers { permanent_uuid: "f3a563398cf94de0ae7d1c748d99faab" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 33807 } }
I20260808 04:07:37.871794 28921 leader_election.cc:288] T 545b8ba3bbe24ec988ba8664704ffee7 P 87132debb51140cb9d97a17d217b6852 [CANDIDATE]: Term 1 election: Requested vote from peers b4fff30c1f1d4b2ab709870b18a9f6e2 (127.5.83.1:34443), f3a563398cf94de0ae7d1c748d99faab (127.5.83.2:33807)
I20260808 04:07:37.872123 28409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "545b8ba3bbe24ec988ba8664704ffee7" candidate_uuid: "87132debb51140cb9d97a17d217b6852" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4fff30c1f1d4b2ab709870b18a9f6e2"
I20260808 04:07:37.872277 28409 raft_consensus.cc:3037] T 545b8ba3bbe24ec988ba8664704ffee7 P b4fff30c1f1d4b2ab709870b18a9f6e2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:07:37.872246 28606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "545b8ba3bbe24ec988ba8664704ffee7" candidate_uuid: "87132debb51140cb9d97a17d217b6852" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3a563398cf94de0ae7d1c748d99faab"
I20260808 04:07:37.872411 28606 raft_consensus.cc:3037] T 545b8ba3bbe24ec988ba8664704ffee7 P f3a563398cf94de0ae7d1c748d99faab [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:07:37.874327 28409 raft_consensus.cc:2454] T 545b8ba3bbe24ec988ba8664704ffee7 P b4fff30c1f1d4b2ab709870b18a9f6e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87132debb51140cb9d97a17d217b6852 in term 1.
I20260808 04:07:37.874513 28606 raft_consensus.cc:2454] T 545b8ba3bbe24ec988ba8664704ffee7 P f3a563398cf94de0ae7d1c748d99faab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87132debb51140cb9d97a17d217b6852 in term 1.
I20260808 04:07:37.874697 28711 leader_election.cc:302] T 545b8ba3bbe24ec988ba8664704ffee7 P 87132debb51140cb9d97a17d217b6852 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87132debb51140cb9d97a17d217b6852, b4fff30c1f1d4b2ab709870b18a9f6e2; no voters:
I20260808 04:07:37.874998 28921 raft_consensus.cc:2781] T 545b8ba3bbe24ec988ba8664704ffee7 P 87132debb51140cb9d97a17d217b6852 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:07:37.875443 28921 raft_consensus.cc:686] T 545b8ba3bbe24ec988ba8664704ffee7 P 87132debb51140cb9d97a17d217b6852 [term 1 LEADER]: Becoming Leader. State: Replica: 87132debb51140cb9d97a17d217b6852, State: Running, Role: LEADER
I20260808 04:07:37.875891 28921 consensus_queue.cc:237] T 545b8ba3bbe24ec988ba8664704ffee7 P 87132debb51140cb9d97a17d217b6852 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87132debb51140cb9d97a17d217b6852" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 42143 } } peers { permanent_uuid: "b4fff30c1f1d4b2ab709870b18a9f6e2" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 34443 } } peers { permanent_uuid: "f3a563398cf94de0ae7d1c748d99faab" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 33807 } }
I20260808 04:07:37.880223 28191 catalog_manager.cc:5495] T 545b8ba3bbe24ec988ba8664704ffee7 P 87132debb51140cb9d97a17d217b6852 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87132debb51140cb9d97a17d217b6852 (127.5.83.3). New cstate: current_term: 1 leader_uuid: "87132debb51140cb9d97a17d217b6852" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87132debb51140cb9d97a17d217b6852" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 42143 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4fff30c1f1d4b2ab709870b18a9f6e2" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 34443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3a563398cf94de0ae7d1c748d99faab" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 33807 } health_report { overall_health: UNKNOWN } } }
I20260808 04:07:37.895326 28908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:37.890131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53510 (local address 127.5.83.3:42143)
0808 04:07:37.890297 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0808 04:07:37.890303 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:07:37.890356 (+ 53us) server_negotiation.cc:407] Connection header received
0808 04:07:37.890569 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:07:37.890572 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:07:37.890641 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:07:37.890758 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:07:37.891136 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:37.891909 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:37.893336 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:37.893735 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:37.893842 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:37.894315 (+ 473us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:07:37.894780 (+ 465us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:07:37.894900 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0808 04:07:37.895017 (+ 117us) server_negotiation.cc:299] Negotiation successful
0808 04:07:37.895091 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 04:07:37.902597 28409 raft_consensus.cc:1261] T 545b8ba3bbe24ec988ba8664704ffee7 P b4fff30c1f1d4b2ab709870b18a9f6e2 [term 1 FOLLOWER]: Refusing update from remote peer 87132debb51140cb9d97a17d217b6852: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:07:37.902598 28606 raft_consensus.cc:1261] T 545b8ba3bbe24ec988ba8664704ffee7 P f3a563398cf94de0ae7d1c748d99faab [term 1 FOLLOWER]: Refusing update from remote peer 87132debb51140cb9d97a17d217b6852: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:07:37.903427 28926 consensus_queue.cc:1035] T 545b8ba3bbe24ec988ba8664704ffee7 P 87132debb51140cb9d97a17d217b6852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4fff30c1f1d4b2ab709870b18a9f6e2" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 34443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:07:37.903790 28921 consensus_queue.cc:1035] T 545b8ba3bbe24ec988ba8664704ffee7 P 87132debb51140cb9d97a17d217b6852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3a563398cf94de0ae7d1c748d99faab" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:07:37.914743 28931 mvcc.cc:205] Tried to move back new op lower bound from 7316119789159985152 to 7316119789063049216. Current Snapshot: MvccSnapshot[applied={T|T < 7316119789159985152}]
I20260808 04:07:37.915731 28933 mvcc.cc:205] Tried to move back new op lower bound from 7316119789159985152 to 7316119789063049216. Current Snapshot: MvccSnapshot[applied={T|T < 7316119789159985152}]
I20260808 04:07:37.927744 28193 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42916:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:07:37.927958 28193 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42916:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:07:37.931231 28193 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ce0c76f0336e4437aa940b8e18eedbab: Sending DeleteTablet for 3 replicas of tablet 545b8ba3bbe24ec988ba8664704ffee7
I20260808 04:07:37.932325 28778 tablet_service.cc:1514] Processing DeleteTablet for tablet 545b8ba3bbe24ec988ba8664704ffee7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:07:37 UTC) from {username='kudu', principal='kudu/127.5.83.62@KRBTEST.COM'} at 127.0.0.1:53496
I20260808 04:07:37.932335 28381 tablet_service.cc:1514] Processing DeleteTablet for tablet 545b8ba3bbe24ec988ba8664704ffee7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:07:37 UTC) from {username='kudu', principal='kudu/127.5.83.62@KRBTEST.COM'} at 127.0.0.1:52148
I20260808 04:07:37.932720 28573 tablet_service.cc:1514] Processing DeleteTablet for tablet 545b8ba3bbe24ec988ba8664704ffee7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:07:37 UTC) from {username='kudu', principal='kudu/127.5.83.62@KRBTEST.COM'} at 127.0.0.1:41510
I20260808 04:07:37.932880 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 28288
I20260808 04:07:37.945427 28945 tablet_replica.cc:330] T 545b8ba3bbe24ec988ba8664704ffee7 P 87132debb51140cb9d97a17d217b6852: stopping tablet replica
I20260808 04:07:37.946022 28945 raft_consensus.cc:2229] T 545b8ba3bbe24ec988ba8664704ffee7 P 87132debb51140cb9d97a17d217b6852 [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:07:37.946777 28947 tablet_replica.cc:330] T 545b8ba3bbe24ec988ba8664704ffee7 P b4fff30c1f1d4b2ab709870b18a9f6e2: stopping tablet replica
I20260808 04:07:37.947077 28947 raft_consensus.cc:2229] T 545b8ba3bbe24ec988ba8664704ffee7 P b4fff30c1f1d4b2ab709870b18a9f6e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:07:37.947471 28947 raft_consensus.cc:2258] T 545b8ba3bbe24ec988ba8664704ffee7 P b4fff30c1f1d4b2ab709870b18a9f6e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:07:37.948168 28945 raft_consensus.cc:2258] T 545b8ba3bbe24ec988ba8664704ffee7 P 87132debb51140cb9d97a17d217b6852 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:07:37.949280 28949 tablet_replica.cc:330] T 545b8ba3bbe24ec988ba8664704ffee7 P f3a563398cf94de0ae7d1c748d99faab: stopping tablet replica
I20260808 04:07:37.949651 28949 raft_consensus.cc:2229] T 545b8ba3bbe24ec988ba8664704ffee7 P f3a563398cf94de0ae7d1c748d99faab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:07:37.950165 28949 raft_consensus.cc:2258] T 545b8ba3bbe24ec988ba8664704ffee7 P f3a563398cf94de0ae7d1c748d99faab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:07:37.950205 28945 ts_tablet_manager.cc:1905] T 545b8ba3bbe24ec988ba8664704ffee7 P 87132debb51140cb9d97a17d217b6852: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:07:37.952291 28949 ts_tablet_manager.cc:1905] T 545b8ba3bbe24ec988ba8664704ffee7 P f3a563398cf94de0ae7d1c748d99faab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:07:37.953051 28947 ts_tablet_manager.cc:1905] T 545b8ba3bbe24ec988ba8664704ffee7 P b4fff30c1f1d4b2ab709870b18a9f6e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:07:37.959180 28949 ts_tablet_manager.cc:1918] T 545b8ba3bbe24ec988ba8664704ffee7 P f3a563398cf94de0ae7d1c748d99faab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:07:37.959352 28949 log.cc:1196] T 545b8ba3bbe24ec988ba8664704ffee7 P f3a563398cf94de0ae7d1c748d99faab: Deleting WAL directory at /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-1/wal/wals/545b8ba3bbe24ec988ba8664704ffee7
I20260808 04:07:37.959928 28949 ts_tablet_manager.cc:1939] T 545b8ba3bbe24ec988ba8664704ffee7 P f3a563398cf94de0ae7d1c748d99faab: Deleting consensus metadata
Aug 08 04:07:37 dist-test-slave-1l3l krb5kdc[28123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162055, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.1@KRBTEST.COM
I20260808 04:07:37.961939 28170 catalog_manager.cc:4841] TS f3a563398cf94de0ae7d1c748d99faab (127.5.83.2:33807): tablet 545b8ba3bbe24ec988ba8664704ffee7 (table test-table [id=7892f64c9b8d4945945d765f566e46a8]) successfully deleted
I20260808 04:07:37.962551 28945 ts_tablet_manager.cc:1918] T 545b8ba3bbe24ec988ba8664704ffee7 P 87132debb51140cb9d97a17d217b6852: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:07:37.962733 28945 log.cc:1196] T 545b8ba3bbe24ec988ba8664704ffee7 P 87132debb51140cb9d97a17d217b6852: Deleting WAL directory at /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-2/wal/wals/545b8ba3bbe24ec988ba8664704ffee7
I20260808 04:07:37.963299 28945 ts_tablet_manager.cc:1939] T 545b8ba3bbe24ec988ba8664704ffee7 P 87132debb51140cb9d97a17d217b6852: Deleting consensus metadata
I20260808 04:07:37.965157 28171 catalog_manager.cc:4841] TS 87132debb51140cb9d97a17d217b6852 (127.5.83.3:42143): tablet 545b8ba3bbe24ec988ba8664704ffee7 (table test-table [id=7892f64c9b8d4945945d765f566e46a8]) successfully deleted
I20260808 04:07:37.967723 28947 ts_tablet_manager.cc:1918] T 545b8ba3bbe24ec988ba8664704ffee7 P b4fff30c1f1d4b2ab709870b18a9f6e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:07:37.967972 28947 log.cc:1196] T 545b8ba3bbe24ec988ba8664704ffee7 P b4fff30c1f1d4b2ab709870b18a9f6e2: Deleting WAL directory at /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-0/wal/wals/545b8ba3bbe24ec988ba8664704ffee7
I20260808 04:07:37.968492 28947 ts_tablet_manager.cc:1939] T 545b8ba3bbe24ec988ba8664704ffee7 P b4fff30c1f1d4b2ab709870b18a9f6e2: Deleting consensus metadata
I20260808 04:07:37.969091 28909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:37.934238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52156 (local address 127.5.83.1:34443)
0808 04:07:37.934419 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0808 04:07:37.934426 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:07:37.949015 (+ 14589us) server_negotiation.cc:407] Connection header received
0808 04:07:37.949247 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:07:37.949253 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:07:37.949357 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:07:37.949486 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:07:37.952181 (+ 2695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:37.953253 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:37.955260 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:37.955920 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:37.956022 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:37.962013 (+ 5991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:07:37.962029 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:07:37.962044 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:07:37.962090 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:07:37.966056 (+ 3966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:37.966751 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:37.966758 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:37.966761 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:37.966887 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:37.967269 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:37.967273 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:37.967275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:37.967652 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:07:37.967994 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0808 04:07:37.968429 (+ 435us) server_negotiation.cc:299] Negotiation successful
0808 04:07:37.968733 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 04:07:37.969780 28469 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:07:37.970328 28168 catalog_manager.cc:4841] TS b4fff30c1f1d4b2ab709870b18a9f6e2 (127.5.83.1:34443): tablet 545b8ba3bbe24ec988ba8664704ffee7 (table test-table [id=7892f64c9b8d4945945d765f566e46a8]) successfully deleted
I20260808 04:07:38.060221 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 28288
I20260808 04:07:38.090163 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 28483
Aug 08 04:07:38 dist-test-slave-1l3l krb5kdc[28123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162055, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.2@KRBTEST.COM
I20260808 04:07:38.109277 28912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:38.091021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41520 (local address 127.5.83.2:33807)
0808 04:07:38.091183 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0808 04:07:38.091188 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:07:38.091206 (+ 18us) server_negotiation.cc:407] Connection header received
0808 04:07:38.091312 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:07:38.091316 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:07:38.091381 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:07:38.091465 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:07:38.093090 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:38.093890 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:38.095810 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:38.096439 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:38.096556 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:38.101960 (+ 5404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:07:38.101980 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:07:38.101999 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:07:38.102045 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:07:38.105793 (+ 3748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:38.106624 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:38.106636 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:38.106640 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:38.106800 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:38.107217 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:38.107222 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:38.107226 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:38.108042 (+ 816us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:07:38.108361 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0808 04:07:38.108684 (+ 323us) server_negotiation.cc:299] Negotiation successful
0808 04:07:38.108919 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 04:07:38.109992 28672 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:07:38.252903 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 28483
I20260808 04:07:38.271909 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 28684
Aug 08 04:07:38 dist-test-slave-1l3l krb5kdc[28123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162055, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.3@KRBTEST.COM
I20260808 04:07:38.288478 28908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:38.272677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44092 (local address 127.5.83.3:42143)
0808 04:07:38.272866 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0808 04:07:38.272871 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:07:38.272940 (+ 69us) server_negotiation.cc:407] Connection header received
0808 04:07:38.273136 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:07:38.273141 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:07:38.273224 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:07:38.273309 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:07:38.274639 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:38.275479 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:38.276951 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:38.277413 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:38.277521 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:38.281875 (+ 4354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:07:38.281887 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:07:38.281903 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:07:38.281939 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:07:38.285098 (+ 3159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:38.285833 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:38.285844 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:38.285847 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:38.286059 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:38.286451 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:38.286457 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:38.286460 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:38.286974 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:07:38.287330 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0808 04:07:38.287551 (+ 221us) server_negotiation.cc:299] Negotiation successful
0808 04:07:38.287846 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260808 04:07:38.289348 28879 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:07:38.411514 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 28684
I20260808 04:07:38.435875 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 28146
I20260808 04:07:38.437036 28244 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:07:38.545341 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 28146
I20260808 04:07:58.568775 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:38749
--webserver_interface=127.5.83.62
--webserver_port=39963
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab
--principal=kudu/127.5.83.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:36095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.62:38749
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:07:58.890337 28958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:07:58.890619 28958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:07:58.890761 28958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:07:58.897676 28958 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260808 04:07:58.897776 28958 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260808 04:07:58.897835 28958 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260808 04:07:58.897913 28958 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:07:58.897989 28958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:07:58.898084 28958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:07:58.898144 28958 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:07:58.898219 28958 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:07:58.907552 28958 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:36095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.5.83.62:38749
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab
--principal=kudu/127.5.83.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:38749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.62
--webserver_port=39963
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:07:58.908754 28958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:07:58.909996 28958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:07:58.932287 28966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:07:58.932204 28971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:07:58.932233 28967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:07:58.933050 28958 server_base.cc:1034] running on GCE node
I20260808 04:07:58.933861 28958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:07:58.935400 28958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:07:58.936663 28958 hybrid_clock.cc:648] HybridClock initialized: now 1786162078936579 us; error 48 us; skew 500 ppm
I20260808 04:07:58.937023 28958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:07:58 dist-test-slave-1l3l krb5kdc[28123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162078, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:07:58.940546 28958 init.cc:377] Logged in from keytab as kudu/127.5.83.62@KRBTEST.COM (short username kudu)
I20260808 04:07:58.942078 28958 webserver.cc:460] Webserver started at http://127.5.83.62:39963/ using document root <none> and password file <none>
I20260808 04:07:58.942754 28958 fs_manager.cc:359] Metadata directory not provided
I20260808 04:07:58.942857 28958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:07:58.948484 28958 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.002s
I20260808 04:07:58.951165 28983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:07:58.952433 28958 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 04:07:58.952575 28958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/master-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/master-0/wal
uuid: "ce0c76f0336e4437aa940b8e18eedbab"
format_stamp: "Formatted at 2026-08-08 04:07:36 on dist-test-slave-1l3l"
server_key: "f62e7a163e755414e6350ee9b8ff99f1"
server_key_iv: "2230815dfdcb2d9d111e02abc6595974"
server_key_version: "encryptionkey@0"
I20260808 04:07:58.953536 28958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:07:58.975521 28958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:07:58.979266 28958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:07:58.979538 28958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:07:58.997092 28958 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.62:38749
I20260808 04:07:58.997128 29063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.62:38749 every 8 connection(s)
I20260808 04:07:58.998510 28958 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
I20260808 04:07:59.005021 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 28958
I20260808 04:07:59.005201 29066 sys_catalog.cc:263] Verifying existing consensus state
I20260808 04:07:59.005846 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.1:34443
--local_ip_for_outbound_sockets=127.5.83.1
--tserver_master_addrs=127.5.83.62:38749
--webserver_port=46157
--webserver_interface=127.5.83.1
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.1.keytab
--principal=kudu/127.5.83.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:36095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:07:59.008430 29066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce0c76f0336e4437aa940b8e18eedbab: Bootstrap starting.
I20260808 04:07:59.028517 29066 log.cc:824] T 00000000000000000000000000000000 P ce0c76f0336e4437aa940b8e18eedbab: Log is configured to *not* fsync() on all Append() calls
I20260808 04:07:59.042188 29066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce0c76f0336e4437aa940b8e18eedbab: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 04:07:59.042776 29066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce0c76f0336e4437aa940b8e18eedbab: Bootstrap complete.
I20260808 04:07:59.051797 29066 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce0c76f0336e4437aa940b8e18eedbab [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce0c76f0336e4437aa940b8e18eedbab" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 38749 } }
I20260808 04:07:59.052601 29066 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce0c76f0336e4437aa940b8e18eedbab [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce0c76f0336e4437aa940b8e18eedbab, State: Initialized, Role: FOLLOWER
I20260808 04:07:59.053167 29066 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce0c76f0336e4437aa940b8e18eedbab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce0c76f0336e4437aa940b8e18eedbab" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 38749 } }
I20260808 04:07:59.053390 29066 raft_consensus.cc:388] T 00000000000000000000000000000000 P ce0c76f0336e4437aa940b8e18eedbab [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:07:59.053499 29066 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce0c76f0336e4437aa940b8e18eedbab [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:07:59.053656 29066 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce0c76f0336e4437aa940b8e18eedbab [term 1 FOLLOWER]: Advancing to term 2
I20260808 04:07:59.058561 29066 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce0c76f0336e4437aa940b8e18eedbab [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce0c76f0336e4437aa940b8e18eedbab" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 38749 } }
I20260808 04:07:59.059072 29066 leader_election.cc:302] T 00000000000000000000000000000000 P ce0c76f0336e4437aa940b8e18eedbab [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce0c76f0336e4437aa940b8e18eedbab; no voters:
I20260808 04:07:59.059731 29066 leader_election.cc:288] T 00000000000000000000000000000000 P ce0c76f0336e4437aa940b8e18eedbab [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 04:07:59.059859 29075 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce0c76f0336e4437aa940b8e18eedbab [term 2 FOLLOWER]: Leader election won for term 2
I20260808 04:07:59.061937 29066 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce0c76f0336e4437aa940b8e18eedbab [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:07:59.066465 29075 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce0c76f0336e4437aa940b8e18eedbab [term 2 LEADER]: Becoming Leader. State: Replica: ce0c76f0336e4437aa940b8e18eedbab, State: Running, Role: LEADER
I20260808 04:07:59.067016 29075 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce0c76f0336e4437aa940b8e18eedbab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce0c76f0336e4437aa940b8e18eedbab" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 38749 } }
I20260808 04:07:59.073388 29079 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce0c76f0336e4437aa940b8e18eedbab [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce0c76f0336e4437aa940b8e18eedbab. Latest consensus state: current_term: 2 leader_uuid: "ce0c76f0336e4437aa940b8e18eedbab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce0c76f0336e4437aa940b8e18eedbab" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 38749 } } }
I20260808 04:07:59.073465 29076 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce0c76f0336e4437aa940b8e18eedbab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ce0c76f0336e4437aa940b8e18eedbab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce0c76f0336e4437aa940b8e18eedbab" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 38749 } } }
I20260808 04:07:59.073845 29076 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce0c76f0336e4437aa940b8e18eedbab [sys.catalog]: This master's current role is: LEADER
I20260808 04:07:59.073890 29079 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce0c76f0336e4437aa940b8e18eedbab [sys.catalog]: This master's current role is: LEADER
I20260808 04:07:59.084656 29093 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:07:59.090732 29093 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:07:59.091816 29093 catalog_manager.cc:1259] Loaded cluster ID: 24183d2003cc437ca80298fbee1ead54
I20260808 04:07:59.091913 29093 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:07:59.094570 29093 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:07:59.095661 29093 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ce0c76f0336e4437aa940b8e18eedbab: Loaded TSK: 0
I20260808 04:07:59.115974 29093 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ce0c76f0336e4437aa940b8e18eedbab: Generated new TSK 1
I20260808 04:07:59.116297 29093 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 04:07:59.353228 29070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:07:59.353488 29070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:07:59.353605 29070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:07:59.360833 29070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:07:59.360931 29070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:07:59.361061 29070 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.1
I20260808 04:07:59.370559 29070 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:36095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.1.keytab
--principal=kudu/127.5.83.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.1:34443
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.83.1
--webserver_port=46157
--tserver_master_addrs=127.5.83.62:38749
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.1
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:07:59.371793 29070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:07:59.373041 29070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:07:59.387065 29110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:07:59.387019 29112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:07:59.387614 29115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:07:59.388674 29070 server_base.cc:1034] running on GCE node
I20260808 04:07:59.389281 29070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:07:59.390221 29070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:07:59.391438 29070 hybrid_clock.cc:648] HybridClock initialized: now 1786162079391389 us; error 44 us; skew 500 ppm
I20260808 04:07:59.391772 29070 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:07:59 dist-test-slave-1l3l krb5kdc[28123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162079, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:07:59.395043 29070 init.cc:377] Logged in from keytab as kudu/127.5.83.1@KRBTEST.COM (short username kudu)
I20260808 04:07:59.396517 29070 webserver.cc:460] Webserver started at http://127.5.83.1:46157/ using document root <none> and password file <none>
I20260808 04:07:59.397173 29070 fs_manager.cc:359] Metadata directory not provided
I20260808 04:07:59.397306 29070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:07:59.402608 29070 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260808 04:07:59.405220 29125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:07:59.406488 29070 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 04:07:59.406637 29070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-0/wal
uuid: "b4fff30c1f1d4b2ab709870b18a9f6e2"
format_stamp: "Formatted at 2026-08-08 04:07:36 on dist-test-slave-1l3l"
server_key: "1305c3b5cdd664233a6c287c4b3d1a5c"
server_key_iv: "700083247ef2d76dd3a98c329e121f88"
server_key_version: "encryptionkey@0"
I20260808 04:07:59.407549 29070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:07:59.421608 29070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:07:59.425611 29070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:07:59.425881 29070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:07:59.426852 29070 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:07:59.428164 29070 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:07:59.428262 29070 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:07:59.428364 29070 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:07:59.428447 29070 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:07:59.461426 29070 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.1:34443
I20260808 04:07:59.461493 29299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.1:34443 every 8 connection(s)
I20260808 04:07:59.462919 29070 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
I20260808 04:07:59.473151 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 29070
I20260808 04:07:59.473810 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.2:33807
--local_ip_for_outbound_sockets=127.5.83.2
--tserver_master_addrs=127.5.83.62:38749
--webserver_port=39145
--webserver_interface=127.5.83.2
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.2.keytab
--principal=kudu/127.5.83.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:36095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 04:07:59 dist-test-slave-1l3l krb5kdc[28123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162079, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.1@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:07:59.486393 29304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:59.465719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.1:42371 (local address 127.5.83.62:38749)
0808 04:07:59.466272 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0808 04:07:59.466288 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 04:07:59.466888 (+ 600us) server_negotiation.cc:407] Connection header received
0808 04:07:59.469125 (+ 2237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:07:59.469138 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:07:59.469448 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:07:59.469758 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:07:59.471267 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:59.472567 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:59.474163 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:07:59.475119 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:07:59.475335 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:59.478852 (+ 3517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:07:59.478862 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:07:59.478878 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:07:59.478928 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:07:59.482457 (+ 3529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:59.483191 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:59.483199 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:59.483206 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:59.483398 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:07:59.483806 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:07:59.483816 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:07:59.483819 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:07:59.484359 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:07:59.484546 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0808 04:07:59.485243 (+ 697us) server_negotiation.cc:299] Negotiation successful
0808 04:07:59.485544 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":244,"threads_started":1}
I20260808 04:07:59.486671 29305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:07:59.465942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:38749 (local address 127.5.83.1:42371)
0808 04:07:59.466721 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0808 04:07:59.466757 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0808 04:07:59.468567 (+ 1810us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:07:59.469942 (+ 1375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:07:59.469953 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:07:59.470549 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:07:59.471070 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:07:59.471086 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:07:59.472729 (+ 1643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:07:59.472732 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:07:59.474015 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:07:59.474030 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:07:59.475380 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:07:59.475386 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:07:59.475588 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:07:59.476291 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:07:59.476314 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:07:59.478553 (+ 2239us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:07:59.482606 (+ 4053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:07:59.482612 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:07:59.482622 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:07:59.483068 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:07:59.483500 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:07:59.483503 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:07:59.483506 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:07:59.483684 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:07:59.484506 (+ 822us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:07:59.484511 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:07:59.484779 (+ 268us) client_negotiation.cc:769] Sending connection context
0808 04:07:59.485003 (+ 224us) client_negotiation.cc:240] Negotiation successful
0808 04:07:59.485466 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":337,"threads_started":1}
I20260808 04:07:59.489355 29301 heartbeater.cc:347] Connected to a master server at 127.5.83.62:38749
I20260808 04:07:59.489650 29301 heartbeater.cc:464] Registering TS with master...
I20260808 04:07:59.490494 29301 heartbeater.cc:511] Master 127.5.83.62:38749 requested a full tablet report, sending...
I20260808 04:07:59.492655 29006 ts_manager.cc:194] Registered new tserver with Master: b4fff30c1f1d4b2ab709870b18a9f6e2 (127.5.83.1:34443)
I20260808 04:07:59.494129 29006 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.1@KRBTEST.COM'} at 127.5.83.1:42371
W20260808 04:07:59.929937 29306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:07:59.930181 29306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:07:59.930313 29306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:07:59.937408 29306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:07:59.937502 29306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:07:59.937649 29306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.2
I20260808 04:07:59.946720 29306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:36095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.2.keytab
--principal=kudu/127.5.83.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.2:33807
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.83.2
--webserver_port=39145
--tserver_master_addrs=127.5.83.62:38749
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.2
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:07:59.947909 29306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:07:59.949303 29306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:07:59.961493 29313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:07:59.961583 29318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:07:59.961526 29311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:07:59.966665 29306 server_base.cc:1034] running on GCE node
I20260808 04:07:59.967278 29306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:07:59.968205 29306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:07:59.969425 29306 hybrid_clock.cc:648] HybridClock initialized: now 1786162079969365 us; error 46 us; skew 500 ppm
I20260808 04:07:59.969717 29306 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:07:59 dist-test-slave-1l3l krb5kdc[28123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162079, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:07:59.972679 29306 init.cc:377] Logged in from keytab as kudu/127.5.83.2@KRBTEST.COM (short username kudu)
I20260808 04:07:59.974076 29306 webserver.cc:460] Webserver started at http://127.5.83.2:39145/ using document root <none> and password file <none>
I20260808 04:07:59.974720 29306 fs_manager.cc:359] Metadata directory not provided
I20260808 04:07:59.974861 29306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:07:59.979996 29306 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260808 04:07:59.982617 29330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:07:59.984092 29306 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:07:59.984236 29306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-1/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-1/wal
uuid: "f3a563398cf94de0ae7d1c748d99faab"
format_stamp: "Formatted at 2026-08-08 04:07:37 on dist-test-slave-1l3l"
server_key: "4509fe7a497d918247ab6a2645d7c1ec"
server_key_iv: "3ae1d35ecc78a2fee8661d6c1b9613d9"
server_key_version: "encryptionkey@0"
I20260808 04:07:59.985181 29306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:08:00.007146 29306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:08:00.011993 29306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:08:00.012265 29306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:08:00.013424 29306 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:08:00.015069 29306 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:08:00.015173 29306 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:08:00.015271 29306 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:08:00.015357 29306 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:08:00.046289 29306 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.2:33807
I20260808 04:08:00.046403 29491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.2:33807 every 8 connection(s)
I20260808 04:08:00.047669 29306 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
I20260808 04:08:00.057124 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 29306
I20260808 04:08:00.057910 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.3:42143
--local_ip_for_outbound_sockets=127.5.83.3
--tserver_master_addrs=127.5.83.62:38749
--webserver_port=34739
--webserver_interface=127.5.83.3
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.3.keytab
--principal=kudu/127.5.83.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:36095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 04:08:00 dist-test-slave-1l3l krb5kdc[28123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162079, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.2@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:08:00.068571 29494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:00.050216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.2:36311 (local address 127.5.83.62:38749)
0808 04:08:00.050654 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:00.050659 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:00.051607 (+ 948us) server_negotiation.cc:407] Connection header received
0808 04:08:00.053431 (+ 1824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:00.053436 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:00.053513 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:00.053611 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:08:00.055072 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:00.055904 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:00.057900 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:00.058767 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:00.058911 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:00.062944 (+ 4033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:08:00.062949 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:08:00.062952 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:08:00.062985 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:08:00.065587 (+ 2602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:08:00.066272 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:08:00.066277 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:08:00.066281 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:08:00.066412 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:08:00.066838 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:08:00.066844 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:08:00.066847 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:08:00.067109 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:08:00.067241 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:00.067941 (+ 700us) server_negotiation.cc:299] Negotiation successful
0808 04:08:00.068159 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":203,"threads_started":1}
I20260808 04:08:00.069535 29495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:00.050747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:38749 (local address 127.5.83.2:36311)
0808 04:08:00.051445 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:00.051489 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 04:08:00.053171 (+ 1682us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:08:00.053860 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:08:00.053868 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:08:00.054364 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:08:00.054889 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:00.054905 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:00.056078 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:00.056082 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:00.057741 (+ 1659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:00.057756 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:00.059097 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:00.059107 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:00.059492 (+ 385us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:00.060407 (+ 915us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:08:00.060465 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:08:00.062709 (+ 2244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:08:00.065725 (+ 3016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:08:00.065734 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:08:00.065749 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:08:00.066151 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:08:00.066530 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:08:00.066535 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:08:00.066539 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:08:00.066721 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:08:00.067241 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:08:00.067249 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:08:00.067555 (+ 306us) client_negotiation.cc:769] Sending connection context
0808 04:08:00.067839 (+ 284us) client_negotiation.cc:240] Negotiation successful
0808 04:08:00.068362 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":288,"threads_started":1}
I20260808 04:08:00.070940 29492 heartbeater.cc:347] Connected to a master server at 127.5.83.62:38749
I20260808 04:08:00.071210 29492 heartbeater.cc:464] Registering TS with master...
I20260808 04:08:00.071955 29492 heartbeater.cc:511] Master 127.5.83.62:38749 requested a full tablet report, sending...
I20260808 04:08:00.073366 29006 ts_manager.cc:194] Registered new tserver with Master: f3a563398cf94de0ae7d1c748d99faab (127.5.83.2:33807)
I20260808 04:08:00.074395 29006 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.2@KRBTEST.COM'} at 127.5.83.2:36311
W20260808 04:08:00.383409 29496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:08:00.383654 29496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:08:00.383790 29496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:08:00.390906 29496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:08:00.391002 29496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:08:00.391119 29496 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.3
I20260808 04:08:00.400529 29496 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:36095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.3.keytab
--principal=kudu/127.5.83.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.3:42143
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.83.3
--webserver_port=34739
--tserver_master_addrs=127.5.83.62:38749
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.3
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:08:00.401644 29496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:08:00.402815 29496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:08:00.415185 29506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:08:00.415148 29507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:08:00.415246 29510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:08:00.416093 29496 server_base.cc:1034] running on GCE node
I20260808 04:08:00.416673 29496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:08:00.417529 29496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:08:00.418730 29496 hybrid_clock.cc:648] HybridClock initialized: now 1786162080418691 us; error 31 us; skew 500 ppm
I20260808 04:08:00.419034 29496 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:08:00 dist-test-slave-1l3l krb5kdc[28123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162080, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:08:00.422062 29496 init.cc:377] Logged in from keytab as kudu/127.5.83.3@KRBTEST.COM (short username kudu)
I20260808 04:08:00.423427 29496 webserver.cc:460] Webserver started at http://127.5.83.3:34739/ using document root <none> and password file <none>
I20260808 04:08:00.424141 29496 fs_manager.cc:359] Metadata directory not provided
I20260808 04:08:00.424270 29496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:08:00.429262 29496 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260808 04:08:00.431869 29519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:08:00.433140 29496 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 04:08:00.433288 29496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-2/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-2/wal
uuid: "87132debb51140cb9d97a17d217b6852"
format_stamp: "Formatted at 2026-08-08 04:07:37 on dist-test-slave-1l3l"
server_key: "5414e96a2625ef619cf6815a5709b6b9"
server_key_iv: "b801ea8a49fd8e0ce02a7c5f1f2dbb1c"
server_key_version: "encryptionkey@0"
I20260808 04:08:00.434299 29496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:08:00.447619 29496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:08:00.451318 29496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:08:00.451579 29496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:08:00.452529 29496 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:08:00.453763 29496 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:08:00.453866 29496 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:08:00.453975 29496 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:08:00.454047 29496 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:08:00.485729 29496 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.3:42143
I20260808 04:08:00.485795 29684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.3:42143 every 8 connection(s)
I20260808 04:08:00.487130 29496 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
I20260808 04:08:00.489321 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 29496
I20260808 04:08:00.497452 29301 heartbeater.cc:503] Master 127.5.83.62:38749 was elected leader, sending a full tablet report...
Aug 08 04:08:00 dist-test-slave-1l3l krb5kdc[28123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162080, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.3@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:08:00.508035 29694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:00.492042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46528 (local address 127.5.83.62:38749)
0808 04:08:00.493540 (+ 1498us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:00.493545 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:00.493562 (+ 17us) server_negotiation.cc:407] Connection header received
0808 04:08:00.493622 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:00.493628 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:00.493698 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:00.493809 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:08:00.495569 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:00.496679 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:00.498481 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:00.498916 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:00.499030 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:00.502288 (+ 3258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:08:00.502293 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:08:00.502296 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:08:00.502324 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:08:00.505392 (+ 3068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:08:00.506043 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:08:00.506049 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:08:00.506053 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:08:00.506221 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:08:00.506575 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:08:00.506580 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:08:00.506584 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:08:00.506937 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:08:00.507196 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:00.507484 (+ 288us) server_negotiation.cc:299] Negotiation successful
0808 04:08:00.507639 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1386,"thread_start_us":1253,"threads_started":1}
I20260808 04:08:00.510454 29494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:00.489876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.3:59059 (local address 127.5.83.62:38749)
0808 04:08:00.490089 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:00.490094 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:00.492314 (+ 2220us) server_negotiation.cc:407] Connection header received
0808 04:08:00.494621 (+ 2307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:00.494626 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:00.494704 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:00.494817 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:08:00.496511 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:00.497664 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:00.499835 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:00.500264 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:00.500359 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:00.504216 (+ 3857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:08:00.504222 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:08:00.504226 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:08:00.504273 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:08:00.507341 (+ 3068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:08:00.508186 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:08:00.508190 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:08:00.508193 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:08:00.508399 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:08:00.508896 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:08:00.508900 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:08:00.508902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:08:00.509150 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:08:00.509279 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:00.510050 (+ 771us) server_negotiation.cc:299] Negotiation successful
0808 04:08:00.510195 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260808 04:08:00.511518 29688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:00.490303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:38749 (local address 127.5.83.3:59059)
0808 04:08:00.492163 (+ 1860us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:00.492207 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 04:08:00.494354 (+ 2147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:08:00.494987 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:08:00.494995 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:08:00.495549 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:08:00.496303 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:00.496325 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:00.497857 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:00.497863 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:00.499643 (+ 1780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:00.499659 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:00.500496 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:00.500504 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:00.500788 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:00.501446 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:08:00.501476 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:08:00.503988 (+ 2512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:08:00.507479 (+ 3491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:08:00.507488 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:08:00.507502 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:08:00.508053 (+ 551us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:08:00.508521 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:08:00.508525 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:08:00.508527 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:08:00.508772 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:08:00.509288 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:08:00.509296 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:08:00.509664 (+ 368us) client_negotiation.cc:769] Sending connection context
0808 04:08:00.509964 (+ 300us) client_negotiation.cc:240] Negotiation successful
0808 04:08:00.510448 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1593,"thread_start_us":1397,"threads_started":1}
I20260808 04:08:00.512827 29685 heartbeater.cc:347] Connected to a master server at 127.5.83.62:38749
I20260808 04:08:00.513098 29685 heartbeater.cc:464] Registering TS with master...
I20260808 04:08:00.513854 29685 heartbeater.cc:511] Master 127.5.83.62:38749 requested a full tablet report, sending...
I20260808 04:08:00.515165 29006 ts_manager.cc:194] Registered new tserver with Master: 87132debb51140cb9d97a17d217b6852 (127.5.83.3:42143)
I20260808 04:08:00.515964 29006 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.3@KRBTEST.COM'} at 127.5.83.3:59059
I20260808 04:08:00.522706 5452 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:08:00.528671 29494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:00.523520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46534 (local address 127.5.83.62:38749)
0808 04:08:00.523730 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:00.523736 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:00.524126 (+ 390us) server_negotiation.cc:407] Connection header received
0808 04:08:00.524268 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:00.524271 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:00.524393 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:00.524507 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:08:00.524854 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:00.525564 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:00.527009 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:00.527393 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:00.527477 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:00.527976 (+ 499us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:08:00.528215 (+ 239us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0808 04:08:00.528479 (+ 264us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46534: token expired
Metrics: {"server-negotiator.queue_time_us":62}
W20260808 04:08:00.528774 29494 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46534: token expired
W20260808 04:08:00.529382 28895 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.5.83.62:38749)
I20260808 04:08:00.541572 29494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:00.529855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46542 (local address 127.5.83.62:38749)
0808 04:08:00.530014 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:00.530019 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:00.530036 (+ 17us) server_negotiation.cc:407] Connection header received
0808 04:08:00.530188 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:00.530191 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:00.530241 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:00.530314 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:08:00.531675 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:00.532368 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:00.533625 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:00.534136 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:00.534243 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:00.536793 (+ 2550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:08:00.536798 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:08:00.536801 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:08:00.536850 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:08:00.539201 (+ 2351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:08:00.539810 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:08:00.539815 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:08:00.539818 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:08:00.539940 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:08:00.540283 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:08:00.540288 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:08:00.540291 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:08:00.540557 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:08:00.540820 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:00.541092 (+ 272us) server_negotiation.cc:299] Negotiation successful
0808 04:08:00.541287 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 04:08:00.544628 29006 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46542:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:08:00.551720 29006 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:08:00.587419 29704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:00.562866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:34443 (local address 127.0.0.1:51088)
0808 04:08:00.578435 (+ 15569us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:00.578454 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 04:08:00.578582 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:08:00.579637 (+ 1055us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:08:00.579645 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:08:00.579679 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:08:00.580150 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:00.580164 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:00.582053 (+ 1889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:00.582057 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:00.584467 (+ 2410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:00.584488 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:00.586442 (+ 1954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:00.586449 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:00.586636 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:00.586694 (+ 58us) client_negotiation.cc:769] Sending connection context
0808 04:08:00.586829 (+ 135us) client_negotiation.cc:240] Negotiation successful
0808 04:08:00.586999 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15444,"mutex_wait_us":28,"thread_start_us":10836,"threads_started":1}
I20260808 04:08:00.587427 29709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:00.564047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51088 (local address 127.5.83.1:34443)
0808 04:08:00.575029 (+ 10982us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:00.575038 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:00.578711 (+ 3673us) server_negotiation.cc:407] Connection header received
0808 04:08:00.578828 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:00.578838 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:00.579127 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:00.579321 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:08:00.580378 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:00.581911 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:00.584634 (+ 2723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:00.585938 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:00.586056 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:00.586290 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:00.586921 (+ 631us) server_negotiation.cc:299] Negotiation successful
0808 04:08:00.587090 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10811,"thread_start_us":10643,"threads_started":1}
I20260808 04:08:00.588879 29706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:00.562647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.3:42143 (local address 127.0.0.1:46758)
0808 04:08:00.574139 (+ 11492us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:00.574190 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 04:08:00.574368 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:08:00.581475 (+ 7107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:08:00.581480 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:08:00.581505 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:08:00.581905 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:00.581917 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:00.584140 (+ 2223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:00.584145 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:00.586190 (+ 2045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:00.586200 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:00.588027 (+ 1827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:00.588033 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:00.588184 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:00.588240 (+ 56us) client_negotiation.cc:769] Sending connection context
0808 04:08:00.588377 (+ 137us) client_negotiation.cc:240] Negotiation successful
0808 04:08:00.588490 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11210,"mutex_wait_us":32,"thread_start_us":15435,"threads_started":1}
I20260808 04:08:00.589098 29708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:00.563511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40776 (local address 127.5.83.2:33807)
0808 04:08:00.580255 (+ 16744us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:00.580265 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:00.581103 (+ 838us) server_negotiation.cc:407] Connection header received
0808 04:08:00.581209 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:00.581216 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:00.581437 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:00.581673 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:08:00.582320 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:00.583455 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:00.586057 (+ 2602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:00.587539 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:00.587663 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:00.587874 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:00.588636 (+ 762us) server_negotiation.cc:299] Negotiation successful
0808 04:08:00.588800 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16598,"thread_start_us":16501,"threads_started":1}
I20260808 04:08:00.589169 29707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:00.563247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.2:33807 (local address 127.0.0.1:40776)
0808 04:08:00.580655 (+ 17408us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:00.580672 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:08:00.580795 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:08:00.581676 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:08:00.581681 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:08:00.581706 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:08:00.582163 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:00.582174 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:00.583584 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:00.583589 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:00.585898 (+ 2309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:00.585913 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:00.587993 (+ 2080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:00.588000 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:00.588182 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:00.588278 (+ 96us) client_negotiation.cc:769] Sending connection context
0808 04:08:00.588349 (+ 71us) client_negotiation.cc:240] Negotiation successful
0808 04:08:00.588471 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17280,"mutex_wait_us":66,"thread_start_us":17117,"threads_started":1}
I20260808 04:08:00.589908 29711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:00.563489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46758 (local address 127.5.83.3:42143)
0808 04:08:00.580709 (+ 17220us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:00.580717 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:00.580750 (+ 33us) server_negotiation.cc:407] Connection header received
0808 04:08:00.580857 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:00.580864 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:00.581091 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:00.582114 (+ 1023us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:08:00.582391 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:00.583982 (+ 1591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:00.586319 (+ 2337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:00.587416 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:00.587970 (+ 554us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:00.588175 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:00.589102 (+ 927us) server_negotiation.cc:299] Negotiation successful
0808 04:08:00.589251 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17054,"thread_start_us":17733,"threads_started":1}
I20260808 04:08:00.591475 29204 tablet_service.cc:1467] Processing CreateTablet for tablet 63ba482b902c4de88778f48a2dc007fc (DEFAULT_TABLE table=test-table [id=945a0f2983e648879f98f2f8066ecaed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:08:00.592715 29401 tablet_service.cc:1467] Processing CreateTablet for tablet 63ba482b902c4de88778f48a2dc007fc (DEFAULT_TABLE table=test-table [id=945a0f2983e648879f98f2f8066ecaed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:08:00.593130 29204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63ba482b902c4de88778f48a2dc007fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:08:00.593763 29401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63ba482b902c4de88778f48a2dc007fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:08:00.593278 29589 tablet_service.cc:1467] Processing CreateTablet for tablet 63ba482b902c4de88778f48a2dc007fc (DEFAULT_TABLE table=test-table [id=945a0f2983e648879f98f2f8066ecaed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:08:00.594549 29589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63ba482b902c4de88778f48a2dc007fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:08:00.605270 29712 tablet_bootstrap.cc:492] T 63ba482b902c4de88778f48a2dc007fc P f3a563398cf94de0ae7d1c748d99faab: Bootstrap starting.
I20260808 04:08:00.607980 29713 tablet_bootstrap.cc:492] T 63ba482b902c4de88778f48a2dc007fc P b4fff30c1f1d4b2ab709870b18a9f6e2: Bootstrap starting.
I20260808 04:08:00.608670 29712 tablet_bootstrap.cc:654] T 63ba482b902c4de88778f48a2dc007fc P f3a563398cf94de0ae7d1c748d99faab: Neither blocks nor log segments found. Creating new log.
I20260808 04:08:00.609375 29716 tablet_bootstrap.cc:492] T 63ba482b902c4de88778f48a2dc007fc P 87132debb51140cb9d97a17d217b6852: Bootstrap starting.
I20260808 04:08:00.610455 29712 log.cc:824] T 63ba482b902c4de88778f48a2dc007fc P f3a563398cf94de0ae7d1c748d99faab: Log is configured to *not* fsync() on all Append() calls
I20260808 04:08:00.612144 29713 tablet_bootstrap.cc:654] T 63ba482b902c4de88778f48a2dc007fc P b4fff30c1f1d4b2ab709870b18a9f6e2: Neither blocks nor log segments found. Creating new log.
I20260808 04:08:00.612546 29716 tablet_bootstrap.cc:654] T 63ba482b902c4de88778f48a2dc007fc P 87132debb51140cb9d97a17d217b6852: Neither blocks nor log segments found. Creating new log.
I20260808 04:08:00.613337 29713 log.cc:824] T 63ba482b902c4de88778f48a2dc007fc P b4fff30c1f1d4b2ab709870b18a9f6e2: Log is configured to *not* fsync() on all Append() calls
I20260808 04:08:00.613826 29716 log.cc:824] T 63ba482b902c4de88778f48a2dc007fc P 87132debb51140cb9d97a17d217b6852: Log is configured to *not* fsync() on all Append() calls
I20260808 04:08:00.614141 29712 tablet_bootstrap.cc:492] T 63ba482b902c4de88778f48a2dc007fc P f3a563398cf94de0ae7d1c748d99faab: No bootstrap required, opened a new log
I20260808 04:08:00.614436 29712 ts_tablet_manager.cc:1397] T 63ba482b902c4de88778f48a2dc007fc P f3a563398cf94de0ae7d1c748d99faab: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260808 04:08:00.616370 29713 tablet_bootstrap.cc:492] T 63ba482b902c4de88778f48a2dc007fc P b4fff30c1f1d4b2ab709870b18a9f6e2: No bootstrap required, opened a new log
I20260808 04:08:00.616643 29713 ts_tablet_manager.cc:1397] T 63ba482b902c4de88778f48a2dc007fc P b4fff30c1f1d4b2ab709870b18a9f6e2: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260808 04:08:00.616768 29716 tablet_bootstrap.cc:492] T 63ba482b902c4de88778f48a2dc007fc P 87132debb51140cb9d97a17d217b6852: No bootstrap required, opened a new log
I20260808 04:08:00.617002 29716 ts_tablet_manager.cc:1397] T 63ba482b902c4de88778f48a2dc007fc P 87132debb51140cb9d97a17d217b6852: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20260808 04:08:00.621567 29712 raft_consensus.cc:348] T 63ba482b902c4de88778f48a2dc007fc P f3a563398cf94de0ae7d1c748d99faab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87132debb51140cb9d97a17d217b6852" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 42143 } } peers { permanent_uuid: "b4fff30c1f1d4b2ab709870b18a9f6e2" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 34443 } } peers { permanent_uuid: "f3a563398cf94de0ae7d1c748d99faab" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 33807 } }
I20260808 04:08:00.621884 29712 raft_consensus.cc:374] T 63ba482b902c4de88778f48a2dc007fc P f3a563398cf94de0ae7d1c748d99faab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:08:00.621949 29712 raft_consensus.cc:729] T 63ba482b902c4de88778f48a2dc007fc P f3a563398cf94de0ae7d1c748d99faab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3a563398cf94de0ae7d1c748d99faab, State: Initialized, Role: FOLLOWER
I20260808 04:08:00.622516 29712 consensus_queue.cc:260] T 63ba482b902c4de88778f48a2dc007fc P f3a563398cf94de0ae7d1c748d99faab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87132debb51140cb9d97a17d217b6852" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 42143 } } peers { permanent_uuid: "b4fff30c1f1d4b2ab709870b18a9f6e2" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 34443 } } peers { permanent_uuid: "f3a563398cf94de0ae7d1c748d99faab" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 33807 } }
I20260808 04:08:00.623600 29492 heartbeater.cc:503] Master 127.5.83.62:38749 was elected leader, sending a full tablet report...
I20260808 04:08:00.624388 29712 ts_tablet_manager.cc:1428] T 63ba482b902c4de88778f48a2dc007fc P f3a563398cf94de0ae7d1c748d99faab: Time spent starting tablet: real 0.010s user 0.001s sys 0.009s
I20260808 04:08:00.627534 29716 raft_consensus.cc:348] T 63ba482b902c4de88778f48a2dc007fc P 87132debb51140cb9d97a17d217b6852 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87132debb51140cb9d97a17d217b6852" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 42143 } } peers { permanent_uuid: "b4fff30c1f1d4b2ab709870b18a9f6e2" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 34443 } } peers { permanent_uuid: "f3a563398cf94de0ae7d1c748d99faab" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 33807 } }
I20260808 04:08:00.628016 29716 raft_consensus.cc:374] T 63ba482b902c4de88778f48a2dc007fc P 87132debb51140cb9d97a17d217b6852 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:08:00.628122 29716 raft_consensus.cc:729] T 63ba482b902c4de88778f48a2dc007fc P 87132debb51140cb9d97a17d217b6852 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87132debb51140cb9d97a17d217b6852, State: Initialized, Role: FOLLOWER
I20260808 04:08:00.628183 29713 raft_consensus.cc:348] T 63ba482b902c4de88778f48a2dc007fc P b4fff30c1f1d4b2ab709870b18a9f6e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87132debb51140cb9d97a17d217b6852" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 42143 } } peers { permanent_uuid: "b4fff30c1f1d4b2ab709870b18a9f6e2" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 34443 } } peers { permanent_uuid: "f3a563398cf94de0ae7d1c748d99faab" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 33807 } }
I20260808 04:08:00.628595 29713 raft_consensus.cc:374] T 63ba482b902c4de88778f48a2dc007fc P b4fff30c1f1d4b2ab709870b18a9f6e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:08:00.628702 29713 raft_consensus.cc:729] T 63ba482b902c4de88778f48a2dc007fc P b4fff30c1f1d4b2ab709870b18a9f6e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4fff30c1f1d4b2ab709870b18a9f6e2, State: Initialized, Role: FOLLOWER
I20260808 04:08:00.628787 29716 consensus_queue.cc:260] T 63ba482b902c4de88778f48a2dc007fc P 87132debb51140cb9d97a17d217b6852 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87132debb51140cb9d97a17d217b6852" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 42143 } } peers { permanent_uuid: "b4fff30c1f1d4b2ab709870b18a9f6e2" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 34443 } } peers { permanent_uuid: "f3a563398cf94de0ae7d1c748d99faab" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 33807 } }
I20260808 04:08:00.629279 29713 consensus_queue.cc:260] T 63ba482b902c4de88778f48a2dc007fc P b4fff30c1f1d4b2ab709870b18a9f6e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87132debb51140cb9d97a17d217b6852" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 42143 } } peers { permanent_uuid: "b4fff30c1f1d4b2ab709870b18a9f6e2" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 34443 } } peers { permanent_uuid: "f3a563398cf94de0ae7d1c748d99faab" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 33807 } }
I20260808 04:08:00.629926 29685 heartbeater.cc:503] Master 127.5.83.62:38749 was elected leader, sending a full tablet report...
I20260808 04:08:00.630638 29716 ts_tablet_manager.cc:1428] T 63ba482b902c4de88778f48a2dc007fc P 87132debb51140cb9d97a17d217b6852: Time spent starting tablet: real 0.014s user 0.003s sys 0.011s
I20260808 04:08:00.630963 29713 ts_tablet_manager.cc:1428] T 63ba482b902c4de88778f48a2dc007fc P b4fff30c1f1d4b2ab709870b18a9f6e2: Time spent starting tablet: real 0.014s user 0.008s sys 0.004s
W20260808 04:08:00.715191 29303 tablet.cc:2369] T 63ba482b902c4de88778f48a2dc007fc P b4fff30c1f1d4b2ab709870b18a9f6e2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 04:08:00.739089 29687 tablet.cc:2369] T 63ba482b902c4de88778f48a2dc007fc P 87132debb51140cb9d97a17d217b6852: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:08:00.770788 29726 raft_consensus.cc:482] T 63ba482b902c4de88778f48a2dc007fc P b4fff30c1f1d4b2ab709870b18a9f6e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:08:00.771075 29726 raft_consensus.cc:504] T 63ba482b902c4de88778f48a2dc007fc P b4fff30c1f1d4b2ab709870b18a9f6e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87132debb51140cb9d97a17d217b6852" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 42143 } } peers { permanent_uuid: "b4fff30c1f1d4b2ab709870b18a9f6e2" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 34443 } } peers { permanent_uuid: "f3a563398cf94de0ae7d1c748d99faab" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 33807 } }
I20260808 04:08:00.772847 29726 leader_election.cc:288] T 63ba482b902c4de88778f48a2dc007fc P b4fff30c1f1d4b2ab709870b18a9f6e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87132debb51140cb9d97a17d217b6852 (127.5.83.3:42143), f3a563398cf94de0ae7d1c748d99faab (127.5.83.2:33807)
I20260808 04:08:00.779534 29708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:00.773490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.1:48121 (local address 127.5.83.2:33807)
0808 04:08:00.773786 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:00.773791 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:00.774422 (+ 631us) server_negotiation.cc:407] Connection header received
0808 04:08:00.774581 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:00.774586 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:00.774662 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:00.774760 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:08:00.775375 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:00.776129 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:00.777868 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:00.778632 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:00.778752 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:00.778846 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:00.779170 (+ 324us) server_negotiation.cc:299] Negotiation successful
0808 04:08:00.779296 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142}
I20260808 04:08:00.779577 29711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:00.773364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.1:39981 (local address 127.5.83.3:42143)
0808 04:08:00.773593 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:00.773600 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:00.774252 (+ 652us) server_negotiation.cc:407] Connection header received
0808 04:08:00.774423 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:00.774429 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:00.774504 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:00.774588 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:08:00.775177 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:00.776006 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:00.777863 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:00.778630 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:00.778744 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:00.778849 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:00.779200 (+ 351us) server_negotiation.cc:299] Negotiation successful
0808 04:08:00.779319 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260808 04:08:00.779631 29728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:00.773089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.3:42143 (local address 127.5.83.1:39981)
0808 04:08:00.774127 (+ 1038us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:00.774151 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0808 04:08:00.774283 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:08:00.774613 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:08:00.774617 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:08:00.774641 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:08:00.775026 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:00.775041 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:00.776144 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:00.776149 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:00.777699 (+ 1550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:00.777711 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:00.778922 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:00.778927 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:00.779049 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:00.779071 (+ 22us) client_negotiation.cc:769] Sending connection context
0808 04:08:00.779148 (+ 77us) client_negotiation.cc:240] Negotiation successful
0808 04:08:00.779265 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":895,"thread_start_us":641,"threads_started":1}
I20260808 04:08:00.780046 29729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:00.773114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.2:33807 (local address 127.5.83.1:48121)
0808 04:08:00.774283 (+ 1169us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:00.774306 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 04:08:00.774457 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:08:00.774786 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:08:00.774789 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:08:00.774809 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:08:00.775214 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:00.775224 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:00.776257 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:00.776261 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:00.777707 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:00.777719 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:00.778869 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:00.778877 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:00.779012 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:00.779036 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 04:08:00.779117 (+ 81us) client_negotiation.cc:240] Negotiation successful
0808 04:08:00.779238 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1059,"mutex_wait_us":1,"thread_start_us":878,"threads_started":1}
I20260808 04:08:00.780619 29621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63ba482b902c4de88778f48a2dc007fc" candidate_uuid: "b4fff30c1f1d4b2ab709870b18a9f6e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87132debb51140cb9d97a17d217b6852" is_pre_election: true
I20260808 04:08:00.781042 29431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63ba482b902c4de88778f48a2dc007fc" candidate_uuid: "b4fff30c1f1d4b2ab709870b18a9f6e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3a563398cf94de0ae7d1c748d99faab" is_pre_election: true
I20260808 04:08:00.781323 29621 raft_consensus.cc:2454] T 63ba482b902c4de88778f48a2dc007fc P 87132debb51140cb9d97a17d217b6852 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4fff30c1f1d4b2ab709870b18a9f6e2 in term 0.
I20260808 04:08:00.781594 29431 raft_consensus.cc:2454] T 63ba482b902c4de88778f48a2dc007fc P f3a563398cf94de0ae7d1c748d99faab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4fff30c1f1d4b2ab709870b18a9f6e2 in term 0.
I20260808 04:08:00.782080 29130 leader_election.cc:302] T 63ba482b902c4de88778f48a2dc007fc P b4fff30c1f1d4b2ab709870b18a9f6e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87132debb51140cb9d97a17d217b6852, b4fff30c1f1d4b2ab709870b18a9f6e2; no voters:
I20260808 04:08:00.782521 29726 raft_consensus.cc:2781] T 63ba482b902c4de88778f48a2dc007fc P b4fff30c1f1d4b2ab709870b18a9f6e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:08:00.782655 29726 raft_consensus.cc:482] T 63ba482b902c4de88778f48a2dc007fc P b4fff30c1f1d4b2ab709870b18a9f6e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:08:00.782753 29726 raft_consensus.cc:3037] T 63ba482b902c4de88778f48a2dc007fc P b4fff30c1f1d4b2ab709870b18a9f6e2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:08:00.784938 29726 raft_consensus.cc:504] T 63ba482b902c4de88778f48a2dc007fc P b4fff30c1f1d4b2ab709870b18a9f6e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87132debb51140cb9d97a17d217b6852" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 42143 } } peers { permanent_uuid: "b4fff30c1f1d4b2ab709870b18a9f6e2" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 34443 } } peers { permanent_uuid: "f3a563398cf94de0ae7d1c748d99faab" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 33807 } }
I20260808 04:08:00.785538 29726 leader_election.cc:288] T 63ba482b902c4de88778f48a2dc007fc P b4fff30c1f1d4b2ab709870b18a9f6e2 [CANDIDATE]: Term 1 election: Requested vote from peers 87132debb51140cb9d97a17d217b6852 (127.5.83.3:42143), f3a563398cf94de0ae7d1c748d99faab (127.5.83.2:33807)
I20260808 04:08:00.785961 29621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63ba482b902c4de88778f48a2dc007fc" candidate_uuid: "b4fff30c1f1d4b2ab709870b18a9f6e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87132debb51140cb9d97a17d217b6852"
I20260808 04:08:00.786038 29431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63ba482b902c4de88778f48a2dc007fc" candidate_uuid: "b4fff30c1f1d4b2ab709870b18a9f6e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3a563398cf94de0ae7d1c748d99faab"
I20260808 04:08:00.786128 29621 raft_consensus.cc:3037] T 63ba482b902c4de88778f48a2dc007fc P 87132debb51140cb9d97a17d217b6852 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:08:00.786229 29431 raft_consensus.cc:3037] T 63ba482b902c4de88778f48a2dc007fc P f3a563398cf94de0ae7d1c748d99faab [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:08:00.788363 29621 raft_consensus.cc:2454] T 63ba482b902c4de88778f48a2dc007fc P 87132debb51140cb9d97a17d217b6852 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4fff30c1f1d4b2ab709870b18a9f6e2 in term 1.
I20260808 04:08:00.788506 29431 raft_consensus.cc:2454] T 63ba482b902c4de88778f48a2dc007fc P f3a563398cf94de0ae7d1c748d99faab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4fff30c1f1d4b2ab709870b18a9f6e2 in term 1.
I20260808 04:08:00.788815 29130 leader_election.cc:302] T 63ba482b902c4de88778f48a2dc007fc P b4fff30c1f1d4b2ab709870b18a9f6e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87132debb51140cb9d97a17d217b6852, b4fff30c1f1d4b2ab709870b18a9f6e2; no voters:
I20260808 04:08:00.789099 29726 raft_consensus.cc:2781] T 63ba482b902c4de88778f48a2dc007fc P b4fff30c1f1d4b2ab709870b18a9f6e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:08:00.789544 29726 raft_consensus.cc:686] T 63ba482b902c4de88778f48a2dc007fc P b4fff30c1f1d4b2ab709870b18a9f6e2 [term 1 LEADER]: Becoming Leader. State: Replica: b4fff30c1f1d4b2ab709870b18a9f6e2, State: Running, Role: LEADER
I20260808 04:08:00.790061 29726 consensus_queue.cc:237] T 63ba482b902c4de88778f48a2dc007fc P b4fff30c1f1d4b2ab709870b18a9f6e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87132debb51140cb9d97a17d217b6852" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 42143 } } peers { permanent_uuid: "b4fff30c1f1d4b2ab709870b18a9f6e2" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 34443 } } peers { permanent_uuid: "f3a563398cf94de0ae7d1c748d99faab" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 33807 } }
I20260808 04:08:00.794225 29006 catalog_manager.cc:5495] T 63ba482b902c4de88778f48a2dc007fc P b4fff30c1f1d4b2ab709870b18a9f6e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to b4fff30c1f1d4b2ab709870b18a9f6e2 (127.5.83.1). New cstate: current_term: 1 leader_uuid: "b4fff30c1f1d4b2ab709870b18a9f6e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87132debb51140cb9d97a17d217b6852" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 42143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4fff30c1f1d4b2ab709870b18a9f6e2" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 34443 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f3a563398cf94de0ae7d1c748d99faab" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 33807 } health_report { overall_health: UNKNOWN } } }
W20260808 04:08:00.799567 29493 tablet.cc:2369] T 63ba482b902c4de88778f48a2dc007fc P f3a563398cf94de0ae7d1c748d99faab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:08:00.821058 29709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:00.815755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51100 (local address 127.5.83.1:34443)
0808 04:08:00.815967 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:00.815973 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:00.815993 (+ 20us) server_negotiation.cc:407] Connection header received
0808 04:08:00.816172 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:00.816176 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:00.816264 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:00.816425 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:08:00.816775 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:00.817494 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:00.818943 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:00.819332 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:00.819446 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:00.820010 (+ 564us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:08:00.820504 (+ 494us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:08:00.820637 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:00.820730 (+ 93us) server_negotiation.cc:299] Negotiation successful
0808 04:08:00.820807 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 04:08:00.828635 29431 raft_consensus.cc:1261] T 63ba482b902c4de88778f48a2dc007fc P f3a563398cf94de0ae7d1c748d99faab [term 1 FOLLOWER]: Refusing update from remote peer b4fff30c1f1d4b2ab709870b18a9f6e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:08:00.828639 29621 raft_consensus.cc:1261] T 63ba482b902c4de88778f48a2dc007fc P 87132debb51140cb9d97a17d217b6852 [term 1 FOLLOWER]: Refusing update from remote peer b4fff30c1f1d4b2ab709870b18a9f6e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:08:00.829514 29731 consensus_queue.cc:1035] T 63ba482b902c4de88778f48a2dc007fc P b4fff30c1f1d4b2ab709870b18a9f6e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87132debb51140cb9d97a17d217b6852" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 42143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:08:00.829895 29726 consensus_queue.cc:1035] T 63ba482b902c4de88778f48a2dc007fc P b4fff30c1f1d4b2ab709870b18a9f6e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3a563398cf94de0ae7d1c748d99faab" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:08:00.845610 29741 mvcc.cc:205] Tried to move back new op lower bound from 7316119883064688640 to 7316119882919256064. Current Snapshot: MvccSnapshot[applied={T|T < 7316119883064688640}]
I20260808 04:08:00.853801 29005 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46542:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:08:00.854148 29005 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46542:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:08:00.858067 29005 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ce0c76f0336e4437aa940b8e18eedbab: Sending DeleteTablet for 3 replicas of tablet 63ba482b902c4de88778f48a2dc007fc
I20260808 04:08:00.859218 29589 tablet_service.cc:1514] Processing DeleteTablet for tablet 63ba482b902c4de88778f48a2dc007fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:08:00 UTC) from {username='kudu', principal='kudu/127.5.83.62@KRBTEST.COM'} at 127.0.0.1:46758
I20260808 04:08:00.859226 29204 tablet_service.cc:1514] Processing DeleteTablet for tablet 63ba482b902c4de88778f48a2dc007fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:08:00 UTC) from {username='kudu', principal='kudu/127.5.83.62@KRBTEST.COM'} at 127.0.0.1:51088
I20260808 04:08:00.859330 29401 tablet_service.cc:1514] Processing DeleteTablet for tablet 63ba482b902c4de88778f48a2dc007fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:08:00 UTC) from {username='kudu', principal='kudu/127.5.83.62@KRBTEST.COM'} at 127.0.0.1:40776
I20260808 04:08:00.860440 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 29070
I20260808 04:08:00.872037 29757 tablet_replica.cc:330] T 63ba482b902c4de88778f48a2dc007fc P b4fff30c1f1d4b2ab709870b18a9f6e2: stopping tablet replica
I20260808 04:08:00.872548 29757 raft_consensus.cc:2229] T 63ba482b902c4de88778f48a2dc007fc P b4fff30c1f1d4b2ab709870b18a9f6e2 [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:08:00.873361 29757 raft_consensus.cc:2258] T 63ba482b902c4de88778f48a2dc007fc P b4fff30c1f1d4b2ab709870b18a9f6e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:08:00.874136 29756 tablet_replica.cc:330] T 63ba482b902c4de88778f48a2dc007fc P 87132debb51140cb9d97a17d217b6852: stopping tablet replica
I20260808 04:08:00.874540 29756 raft_consensus.cc:2229] T 63ba482b902c4de88778f48a2dc007fc P 87132debb51140cb9d97a17d217b6852 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:08:00.875025 29756 raft_consensus.cc:2258] T 63ba482b902c4de88778f48a2dc007fc P 87132debb51140cb9d97a17d217b6852 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:08:00.877432 29756 ts_tablet_manager.cc:1905] T 63ba482b902c4de88778f48a2dc007fc P 87132debb51140cb9d97a17d217b6852: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:08:00.883288 29709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:00.861244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51116 (local address 127.5.83.1:34443)
0808 04:08:00.861418 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:00.861424 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:00.861703 (+ 279us) server_negotiation.cc:407] Connection header received
0808 04:08:00.861768 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:00.861774 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:00.861869 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:00.861955 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:08:00.864045 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:00.865391 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:00.867441 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:00.868220 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:00.868336 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:00.875036 (+ 6700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:08:00.875050 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:08:00.875066 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:08:00.875111 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:08:00.879138 (+ 4027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:08:00.880100 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:08:00.880109 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:08:00.880112 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:08:00.880279 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:08:00.880729 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:08:00.880734 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:08:00.880737 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:08:00.881173 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:08:00.881471 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:00.882656 (+ 1185us) server_negotiation.cc:299] Negotiation successful
0808 04:08:00.882901 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 04:08:00.884248 29295 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:08:00.885378 29756 ts_tablet_manager.cc:1918] T 63ba482b902c4de88778f48a2dc007fc P 87132debb51140cb9d97a17d217b6852: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:08:00.885537 29756 log.cc:1196] T 63ba482b902c4de88778f48a2dc007fc P 87132debb51140cb9d97a17d217b6852: Deleting WAL directory at /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-2/wal/wals/63ba482b902c4de88778f48a2dc007fc
I20260808 04:08:00.886124 29758 tablet_replica.cc:330] T 63ba482b902c4de88778f48a2dc007fc P f3a563398cf94de0ae7d1c748d99faab: stopping tablet replica
I20260808 04:08:00.886169 29756 ts_tablet_manager.cc:1939] T 63ba482b902c4de88778f48a2dc007fc P 87132debb51140cb9d97a17d217b6852: Deleting consensus metadata
I20260808 04:08:00.886552 29758 raft_consensus.cc:2229] T 63ba482b902c4de88778f48a2dc007fc P f3a563398cf94de0ae7d1c748d99faab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:08:00.887080 29758 raft_consensus.cc:2258] T 63ba482b902c4de88778f48a2dc007fc P f3a563398cf94de0ae7d1c748d99faab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:08:00.888180 28989 catalog_manager.cc:4841] TS 87132debb51140cb9d97a17d217b6852 (127.5.83.3:42143): tablet 63ba482b902c4de88778f48a2dc007fc (table test-table [id=945a0f2983e648879f98f2f8066ecaed]) successfully deleted
I20260808 04:08:00.889202 29758 ts_tablet_manager.cc:1905] T 63ba482b902c4de88778f48a2dc007fc P f3a563398cf94de0ae7d1c748d99faab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:08:00.896549 29758 ts_tablet_manager.cc:1918] T 63ba482b902c4de88778f48a2dc007fc P f3a563398cf94de0ae7d1c748d99faab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:08:00.896724 29758 log.cc:1196] T 63ba482b902c4de88778f48a2dc007fc P f3a563398cf94de0ae7d1c748d99faab: Deleting WAL directory at /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-1/wal/wals/63ba482b902c4de88778f48a2dc007fc
I20260808 04:08:00.897253 29758 ts_tablet_manager.cc:1939] T 63ba482b902c4de88778f48a2dc007fc P f3a563398cf94de0ae7d1c748d99faab: Deleting consensus metadata
I20260808 04:08:00.899142 28988 catalog_manager.cc:4841] TS f3a563398cf94de0ae7d1c748d99faab (127.5.83.2:33807): tablet 63ba482b902c4de88778f48a2dc007fc (table test-table [id=945a0f2983e648879f98f2f8066ecaed]) successfully deleted
I20260808 04:08:00.983856 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 29070
W20260808 04:08:01.004810 28986 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.83.1:34443 (error 108)
I20260808 04:08:01.004973 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 29306
I20260808 04:08:01.007010 29707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:01.006566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:34443 (local address 127.0.0.1:51126)
0808 04:08:01.006733 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:01.006848 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.1:34443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260808 04:08:01.007368 28986 catalog_manager.cc:4584] TS b4fff30c1f1d4b2ab709870b18a9f6e2 (127.5.83.1:34443): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 63ba482b902c4de88778f48a2dc007fc: Network error: Client connection negotiation failed: client connection to 127.5.83.1:34443: connect: Connection refused (error 111)
I20260808 04:08:01.019832 29708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:01.005831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40782 (local address 127.5.83.2:33807)
0808 04:08:01.006014 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:01.006020 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:01.006077 (+ 57us) server_negotiation.cc:407] Connection header received
0808 04:08:01.006252 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:01.006257 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:01.006359 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:01.006459 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:08:01.008451 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:01.009612 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:01.011013 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:01.011640 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:01.011808 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:01.014443 (+ 2635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:08:01.014458 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:08:01.014474 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:08:01.014507 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:08:01.017327 (+ 2820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:08:01.018027 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:08:01.018033 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:08:01.018035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:08:01.018152 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:08:01.018529 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:08:01.018532 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:08:01.018534 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:08:01.018895 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:08:01.019109 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:01.019358 (+ 249us) server_negotiation.cc:299] Negotiation successful
0808 04:08:01.019523 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 04:08:01.020493 29486 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:08:01.061388 29707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:01.060758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:34443 (local address 127.0.0.1:51140)
0808 04:08:01.061077 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:01.061195 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.1:34443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186}
I20260808 04:08:01.062906 29707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:01.062366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:34443 (local address 127.0.0.1:51142)
0808 04:08:01.062569 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:01.062648 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.1:34443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260808 04:08:01.117592 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 29306
I20260808 04:08:01.123432 29707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:01.122828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:34443 (local address 127.0.0.1:51156)
0808 04:08:01.123081 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:01.123209 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.1:34443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260808 04:08:01.124557 29707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:01.124193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:34443 (local address 127.0.0.1:51170)
0808 04:08:01.124340 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:01.124427 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.1:34443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260808 04:08:01.140257 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 29496
I20260808 04:08:01.152964 29711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:01.141005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46762 (local address 127.5.83.3:42143)
0808 04:08:01.141159 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:01.141163 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:01.141184 (+ 21us) server_negotiation.cc:407] Connection header received
0808 04:08:01.141397 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:01.141412 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:01.141498 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:01.141585 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:08:01.142900 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:01.143604 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:01.144931 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:01.145321 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:01.145423 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:01.148036 (+ 2613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:08:01.148051 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:08:01.148068 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:08:01.148101 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:08:01.150627 (+ 2526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:08:01.151254 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:08:01.151262 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:08:01.151264 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:08:01.151427 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:08:01.151716 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:08:01.151721 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:08:01.151730 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:08:01.152076 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:08:01.152300 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:01.152560 (+ 260us) server_negotiation.cc:299] Negotiation successful
0808 04:08:01.152713 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 04:08:01.153573 29679 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:08:01.205300 29707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:01.204716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:34443 (local address 127.0.0.1:51176)
0808 04:08:01.204972 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:01.205090 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.1:34443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260808 04:08:01.206457 29707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:01.206021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:34443 (local address 127.0.0.1:51182)
0808 04:08:01.206225 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:01.206325 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.1:34443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260808 04:08:01.244122 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 29496
I20260808 04:08:01.263006 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 28958
I20260808 04:08:01.264127 29055 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:08:01.377583 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 28958
I20260808 04:08:21.403512 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:38749
--webserver_interface=127.5.83.62
--webserver_port=39963
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab
--principal=kudu/127.5.83.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:36095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.62:38749
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:08:21.723996 29764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:08:21.724262 29764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:08:21.724421 29764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:08:21.731360 29764 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260808 04:08:21.731448 29764 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260808 04:08:21.731491 29764 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260808 04:08:21.731560 29764 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:08:21.731627 29764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:08:21.731739 29764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:08:21.731796 29764 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:08:21.731868 29764 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:08:21.741276 29764 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:36095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.5.83.62:38749
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab
--principal=kudu/127.5.83.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:38749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.62
--webserver_port=39963
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:08:21.742334 29764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:08:21.743541 29764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:08:21.754508 29774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:08:21.754477 29780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:08:21.754561 29773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:08:21.755131 29764 server_base.cc:1034] running on GCE node
I20260808 04:08:21.755854 29764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:08:21.757047 29764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:08:21.758252 29764 hybrid_clock.cc:648] HybridClock initialized: now 1786162101758192 us; error 39 us; skew 500 ppm
I20260808 04:08:21.758540 29764 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:08:21 dist-test-slave-1l3l krb5kdc[28123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162101, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:08:21.761523 29764 init.cc:377] Logged in from keytab as kudu/127.5.83.62@KRBTEST.COM (short username kudu)
I20260808 04:08:21.762832 29764 webserver.cc:460] Webserver started at http://127.5.83.62:39963/ using document root <none> and password file <none>
I20260808 04:08:21.763427 29764 fs_manager.cc:359] Metadata directory not provided
I20260808 04:08:21.763513 29764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:08:21.768441 29764 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:08:21.772380 29788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:08:21.773602 29764 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 04:08:21.773739 29764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/master-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/master-0/wal
uuid: "ce0c76f0336e4437aa940b8e18eedbab"
format_stamp: "Formatted at 2026-08-08 04:07:36 on dist-test-slave-1l3l"
server_key: "f62e7a163e755414e6350ee9b8ff99f1"
server_key_iv: "2230815dfdcb2d9d111e02abc6595974"
server_key_version: "encryptionkey@0"
I20260808 04:08:21.774617 29764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:08:21.799902 29764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:08:21.803617 29764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:08:21.803882 29764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:08:21.820802 29764 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.62:38749
I20260808 04:08:21.820843 29857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.62:38749 every 8 connection(s)
I20260808 04:08:21.822180 29764 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
I20260808 04:08:21.828678 29859 sys_catalog.cc:263] Verifying existing consensus state
I20260808 04:08:21.829627 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 29764
I20260808 04:08:21.830674 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.1:34443
--local_ip_for_outbound_sockets=127.5.83.1
--tserver_master_addrs=127.5.83.62:38749
--webserver_port=46157
--webserver_interface=127.5.83.1
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.1.keytab
--principal=kudu/127.5.83.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:36095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:08:21.832954 29859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce0c76f0336e4437aa940b8e18eedbab: Bootstrap starting.
I20260808 04:08:21.866526 29859 log.cc:824] T 00000000000000000000000000000000 P ce0c76f0336e4437aa940b8e18eedbab: Log is configured to *not* fsync() on all Append() calls
I20260808 04:08:21.887012 29859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce0c76f0336e4437aa940b8e18eedbab: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 04:08:21.887642 29859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce0c76f0336e4437aa940b8e18eedbab: Bootstrap complete.
I20260808 04:08:21.900635 29859 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce0c76f0336e4437aa940b8e18eedbab [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce0c76f0336e4437aa940b8e18eedbab" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 38749 } }
I20260808 04:08:21.901924 29859 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce0c76f0336e4437aa940b8e18eedbab [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce0c76f0336e4437aa940b8e18eedbab, State: Initialized, Role: FOLLOWER
I20260808 04:08:21.902690 29859 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce0c76f0336e4437aa940b8e18eedbab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce0c76f0336e4437aa940b8e18eedbab" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 38749 } }
I20260808 04:08:21.902998 29859 raft_consensus.cc:388] T 00000000000000000000000000000000 P ce0c76f0336e4437aa940b8e18eedbab [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:08:21.903141 29859 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce0c76f0336e4437aa940b8e18eedbab [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:08:21.903343 29859 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce0c76f0336e4437aa940b8e18eedbab [term 2 FOLLOWER]: Advancing to term 3
I20260808 04:08:21.909188 29859 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce0c76f0336e4437aa940b8e18eedbab [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce0c76f0336e4437aa940b8e18eedbab" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 38749 } }
I20260808 04:08:21.909780 29859 leader_election.cc:302] T 00000000000000000000000000000000 P ce0c76f0336e4437aa940b8e18eedbab [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce0c76f0336e4437aa940b8e18eedbab; no voters:
I20260808 04:08:21.910497 29859 leader_election.cc:288] T 00000000000000000000000000000000 P ce0c76f0336e4437aa940b8e18eedbab [CANDIDATE]: Term 3 election: Requested vote from peers
I20260808 04:08:21.910620 29865 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce0c76f0336e4437aa940b8e18eedbab [term 3 FOLLOWER]: Leader election won for term 3
I20260808 04:08:21.911613 29865 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce0c76f0336e4437aa940b8e18eedbab [term 3 LEADER]: Becoming Leader. State: Replica: ce0c76f0336e4437aa940b8e18eedbab, State: Running, Role: LEADER
I20260808 04:08:21.912096 29865 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce0c76f0336e4437aa940b8e18eedbab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce0c76f0336e4437aa940b8e18eedbab" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 38749 } }
I20260808 04:08:21.912621 29859 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce0c76f0336e4437aa940b8e18eedbab [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:08:21.914242 29868 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce0c76f0336e4437aa940b8e18eedbab [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce0c76f0336e4437aa940b8e18eedbab. Latest consensus state: current_term: 3 leader_uuid: "ce0c76f0336e4437aa940b8e18eedbab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce0c76f0336e4437aa940b8e18eedbab" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 38749 } } }
I20260808 04:08:21.914602 29868 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce0c76f0336e4437aa940b8e18eedbab [sys.catalog]: This master's current role is: LEADER
I20260808 04:08:21.915117 29866 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce0c76f0336e4437aa940b8e18eedbab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "ce0c76f0336e4437aa940b8e18eedbab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce0c76f0336e4437aa940b8e18eedbab" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 38749 } } }
I20260808 04:08:21.915421 29866 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce0c76f0336e4437aa940b8e18eedbab [sys.catalog]: This master's current role is: LEADER
I20260808 04:08:21.918730 29872 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:08:21.926709 29872 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:08:21.928135 29872 catalog_manager.cc:1259] Loaded cluster ID: 24183d2003cc437ca80298fbee1ead54
I20260808 04:08:21.928277 29872 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:08:21.932207 29872 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:08:21.933782 29872 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ce0c76f0336e4437aa940b8e18eedbab: Loaded TSK: 0
I20260808 04:08:21.933984 29872 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ce0c76f0336e4437aa940b8e18eedbab: Loaded TSK: 1
F20260808 04:08:21.935132 29872 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fcc8db619ad google::LogMessage::Fail() at ??:0
@ 0x7fcc8db6488a google::LogMessage::SendToLog() at ??:0
@ 0x7fcc8db614de google::LogMessage::Flush() at ??:0
@ 0x7fcc8db62a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fcc918c57d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fcc918b473e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fcca35901c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fcca358f783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fcca361b2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7fcca361b0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7fcca36655e9 std::function<>::operator()() at ??:0
@ 0x7fcca3593cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fcca3591c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fcca3619610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fcca3619439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fcc907ad345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fcc907b091c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fcc907b0799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fcc9078beaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fcc93a656db start_thread at ??:0
@ 0x7fcc8c9fb61f clone at ??:0
W20260808 04:08:22.247891 29862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:08:22.248148 29862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:08:22.248255 29862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:08:22.255185 29862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:08:22.255280 29862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:08:22.255414 29862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.1
I20260808 04:08:22.265015 29862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:36095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.1.keytab
--principal=kudu/127.5.83.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.1:34443
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.83.1
--webserver_port=46157
--tserver_master_addrs=127.5.83.62:38749
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.1
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:08:22.266177 29862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:08:22.267396 29862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:08:22.279759 29932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:08:22.279855 29934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:08:22.280083 29931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:08:22.280653 29862 server_base.cc:1034] running on GCE node
I20260808 04:08:22.281178 29862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:08:22.281966 29862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:08:22.283155 29862 hybrid_clock.cc:648] HybridClock initialized: now 1786162102283114 us; error 35 us; skew 500 ppm
I20260808 04:08:22.283435 29862 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:08:22 dist-test-slave-1l3l krb5kdc[28123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162102, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:08:22.286340 29862 init.cc:377] Logged in from keytab as kudu/127.5.83.1@KRBTEST.COM (short username kudu)
I20260808 04:08:22.287640 29862 webserver.cc:460] Webserver started at http://127.5.83.1:46157/ using document root <none> and password file <none>
I20260808 04:08:22.288276 29862 fs_manager.cc:359] Metadata directory not provided
I20260808 04:08:22.288395 29862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:08:22.293275 29862 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.004s
I20260808 04:08:22.295845 29942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:08:22.297032 29862 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 04:08:22.297175 29862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-0/wal
uuid: "b4fff30c1f1d4b2ab709870b18a9f6e2"
format_stamp: "Formatted at 2026-08-08 04:07:36 on dist-test-slave-1l3l"
server_key: "1305c3b5cdd664233a6c287c4b3d1a5c"
server_key_iv: "700083247ef2d76dd3a98c329e121f88"
server_key_version: "encryptionkey@0"
I20260808 04:08:22.298059 29862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:08:22.319501 29862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:08:22.323122 29862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:08:22.323357 29862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:08:22.324302 29862 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:08:22.325977 29949 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260808 04:08:22.331296 29862 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260808 04:08:22.331413 29862 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260808 04:08:22.331521 29862 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260808 04:08:22.334542 29862 ts_tablet_manager.cc:610] Registered 1 tablets
I20260808 04:08:22.334623 29862 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.000s
I20260808 04:08:22.335029 29949 tablet_bootstrap.cc:492] T 63ba482b902c4de88778f48a2dc007fc P b4fff30c1f1d4b2ab709870b18a9f6e2: Bootstrap starting.
I20260808 04:08:22.343317 29949 log.cc:824] T 63ba482b902c4de88778f48a2dc007fc P b4fff30c1f1d4b2ab709870b18a9f6e2: Log is configured to *not* fsync() on all Append() calls
I20260808 04:08:22.357904 29949 tablet_bootstrap.cc:492] T 63ba482b902c4de88778f48a2dc007fc P b4fff30c1f1d4b2ab709870b18a9f6e2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 04:08:22.358763 29949 tablet_bootstrap.cc:492] T 63ba482b902c4de88778f48a2dc007fc P b4fff30c1f1d4b2ab709870b18a9f6e2: Bootstrap complete.
I20260808 04:08:22.359349 29949 ts_tablet_manager.cc:1397] T 63ba482b902c4de88778f48a2dc007fc P b4fff30c1f1d4b2ab709870b18a9f6e2: Time spent bootstrapping tablet: real 0.025s user 0.009s sys 0.011s
I20260808 04:08:22.367941 29949 raft_consensus.cc:348] T 63ba482b902c4de88778f48a2dc007fc P b4fff30c1f1d4b2ab709870b18a9f6e2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87132debb51140cb9d97a17d217b6852" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 42143 } } peers { permanent_uuid: "b4fff30c1f1d4b2ab709870b18a9f6e2" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 34443 } } peers { permanent_uuid: "f3a563398cf94de0ae7d1c748d99faab" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 33807 } }
I20260808 04:08:22.369411 29949 raft_consensus.cc:729] T 63ba482b902c4de88778f48a2dc007fc P b4fff30c1f1d4b2ab709870b18a9f6e2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4fff30c1f1d4b2ab709870b18a9f6e2, State: Initialized, Role: FOLLOWER
I20260808 04:08:22.370127 29949 consensus_queue.cc:260] T 63ba482b902c4de88778f48a2dc007fc P b4fff30c1f1d4b2ab709870b18a9f6e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87132debb51140cb9d97a17d217b6852" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 42143 } } peers { permanent_uuid: "b4fff30c1f1d4b2ab709870b18a9f6e2" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 34443 } } peers { permanent_uuid: "f3a563398cf94de0ae7d1c748d99faab" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 33807 } }
I20260808 04:08:22.371794 29862 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.1:34443
I20260808 04:08:22.371799 29949 ts_tablet_manager.cc:1428] T 63ba482b902c4de88778f48a2dc007fc P b4fff30c1f1d4b2ab709870b18a9f6e2: Time spent starting tablet: real 0.012s user 0.005s sys 0.008s
I20260808 04:08:22.371843 30109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.1:34443 every 8 connection(s)
I20260808 04:08:22.372962 29862 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
I20260808 04:08:22.376362 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 29862
I20260808 04:08:22.377050 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.2:33807
--local_ip_for_outbound_sockets=127.5.83.2
--tserver_master_addrs=127.5.83.62:38749
--webserver_port=39145
--webserver_interface=127.5.83.2
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.2.keytab
--principal=kudu/127.5.83.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:36095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:08:22.377292 30112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:22.375442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:38749 (local address 127.5.83.1:38023)
0808 04:08:22.376249 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:22.376447 (+ 198us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:38749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":399,"threads_started":1}
W20260808 04:08:22.377971 30110 heartbeater.cc:644] Failed to heartbeat to 127.5.83.62:38749 (0 consecutive failures): Network error: Failed to ping master at 127.5.83.62:38749: Client connection negotiation failed: client connection to 127.5.83.62:38749: connect: Connection refused (error 111)
I20260808 04:08:22.378762 30112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:22.378461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:38749 (local address 127.5.83.1:48433)
0808 04:08:22.378605 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:22.378666 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:38749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260808 04:08:22.379539 30112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:22.379247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:38749 (local address 127.5.83.1:43915)
0808 04:08:22.379362 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:22.379429 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:38749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260808 04:08:22.379758 30110 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 04:08:22.694833 30114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:08:22.695077 30114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:08:22.695201 30114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:08:22.702135 30114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:08:22.702225 30114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:08:22.702364 30114 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.2
I20260808 04:08:22.711412 30114 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:36095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.2.keytab
--principal=kudu/127.5.83.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.2:33807
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.83.2
--webserver_port=39145
--tserver_master_addrs=127.5.83.62:38749
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.2
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:08:22.712555 30114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:08:22.713725 30114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:08:22.725955 30122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:08:22.725879 30124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:08:22.725922 30126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:08:22.726755 30114 server_base.cc:1034] running on GCE node
I20260808 04:08:22.727308 30114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:08:22.728240 30114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:08:22.729446 30114 hybrid_clock.cc:648] HybridClock initialized: now 1786162102729393 us; error 40 us; skew 500 ppm
I20260808 04:08:22.729761 30114 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:08:22 dist-test-slave-1l3l krb5kdc[28123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162102, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:08:22.736114 30114 init.cc:377] Logged in from keytab as kudu/127.5.83.2@KRBTEST.COM (short username kudu)
I20260808 04:08:22.737495 30114 webserver.cc:460] Webserver started at http://127.5.83.2:39145/ using document root <none> and password file <none>
I20260808 04:08:22.738117 30114 fs_manager.cc:359] Metadata directory not provided
I20260808 04:08:22.738235 30114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:08:22.743207 30114 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260808 04:08:22.745703 30136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:08:22.747045 30114 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 04:08:22.747195 30114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-1/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-1/wal
uuid: "f3a563398cf94de0ae7d1c748d99faab"
format_stamp: "Formatted at 2026-08-08 04:07:37 on dist-test-slave-1l3l"
server_key: "4509fe7a497d918247ab6a2645d7c1ec"
server_key_iv: "3ae1d35ecc78a2fee8661d6c1b9613d9"
server_key_version: "encryptionkey@0"
I20260808 04:08:22.748159 30114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:08:22.763975 30114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:08:22.767360 30114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:08:22.767596 30114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:08:22.768502 30114 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:08:22.769714 30114 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:08:22.769807 30114 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:08:22.769920 30114 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:08:22.769996 30114 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:08:22.800233 30114 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.2:33807
I20260808 04:08:22.800298 30301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.2:33807 every 8 connection(s)
I20260808 04:08:22.801604 30114 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
I20260808 04:08:22.802568 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 30114
I20260808 04:08:22.803371 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.3:42143
--local_ip_for_outbound_sockets=127.5.83.3
--tserver_master_addrs=127.5.83.62:38749
--webserver_port=34739
--webserver_interface=127.5.83.3
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.3.keytab
--principal=kudu/127.5.83.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:36095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:08:22.806746 30308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:22.804624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:38749 (local address 127.5.83.2:59177)
0808 04:08:22.805449 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:22.805665 (+ 216us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:38749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":329,"threads_started":1}
W20260808 04:08:22.807425 30305 heartbeater.cc:644] Failed to heartbeat to 127.5.83.62:38749 (0 consecutive failures): Network error: Failed to ping master at 127.5.83.62:38749: Client connection negotiation failed: client connection to 127.5.83.62:38749: connect: Connection refused (error 111)
I20260808 04:08:22.808239 30308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:22.807905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:38749 (local address 127.5.83.2:58783)
0808 04:08:22.808046 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:22.808120 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:38749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260808 04:08:22.809106 30308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:22.808757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:38749 (local address 127.5.83.2:49251)
0808 04:08:22.808895 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:22.808982 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:38749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260808 04:08:22.809327 30305 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 04:08:23.132650 30309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:08:23.132905 30309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:08:23.133044 30309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:08:23.140244 30309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:08:23.140353 30309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:08:23.140514 30309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.3
I20260808 04:08:23.149972 30309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:36095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.3.keytab
--principal=kudu/127.5.83.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.3:42143
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.83.3
--webserver_port=34739
--tserver_master_addrs=127.5.83.62:38749
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.3
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:08:23.151245 30309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:08:23.152776 30309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:08:23.165112 30315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:08:23.165144 30318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:08:23.165176 30321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:08:23.165943 30309 server_base.cc:1034] running on GCE node
I20260808 04:08:23.166472 30309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:08:23.167311 30309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:08:23.168540 30309 hybrid_clock.cc:648] HybridClock initialized: now 1786162103168497 us; error 34 us; skew 500 ppm
I20260808 04:08:23.168834 30309 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:08:23 dist-test-slave-1l3l krb5kdc[28123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162103, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:08:23.171813 30309 init.cc:377] Logged in from keytab as kudu/127.5.83.3@KRBTEST.COM (short username kudu)
I20260808 04:08:23.173193 30309 webserver.cc:460] Webserver started at http://127.5.83.3:34739/ using document root <none> and password file <none>
I20260808 04:08:23.173794 30309 fs_manager.cc:359] Metadata directory not provided
I20260808 04:08:23.173916 30309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:08:23.178934 30309 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.004s
I20260808 04:08:23.181459 30331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:08:23.182624 30309 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260808 04:08:23.182775 30309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-2/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-2/wal
uuid: "87132debb51140cb9d97a17d217b6852"
format_stamp: "Formatted at 2026-08-08 04:07:37 on dist-test-slave-1l3l"
server_key: "5414e96a2625ef619cf6815a5709b6b9"
server_key_iv: "b801ea8a49fd8e0ce02a7c5f1f2dbb1c"
server_key_version: "encryptionkey@0"
I20260808 04:08:23.183755 30309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:08:23.203660 30309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:08:23.207166 30309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:08:23.207401 30309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:08:23.208330 30309 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:08:23.209578 30309 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:08:23.209674 30309 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:08:23.209791 30309 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:08:23.209875 30309 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:08:23.241017 30309 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.3:42143
I20260808 04:08:23.241091 30504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.3:42143 every 8 connection(s)
I20260808 04:08:23.242424 30309 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
I20260808 04:08:23.247220 30507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:23.245422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:38749 (local address 127.5.83.3:51373)
0808 04:08:23.246156 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:23.246376 (+ 220us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:38749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":282,"threads_started":1}
W20260808 04:08:23.247856 30505 heartbeater.cc:644] Failed to heartbeat to 127.5.83.62:38749 (0 consecutive failures): Network error: Failed to ping master at 127.5.83.62:38749: Client connection negotiation failed: client connection to 127.5.83.62:38749: connect: Connection refused (error 111)
I20260808 04:08:23.248755 30507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:23.248422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:38749 (local address 127.5.83.3:48349)
0808 04:08:23.248573 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:23.248648 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:38749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260808 04:08:23.249465 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 30309
I20260808 04:08:23.249698 30507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:23.249312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:38749 (local address 127.5.83.3:37509)
0808 04:08:23.249464 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:23.249557 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:38749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260808 04:08:23.249987 30505 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260808 04:08:23.381006 30509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:23.380212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:38749 (local address 127.5.83.1:53667)
0808 04:08:23.380720 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:23.380805 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:38749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":257,"threads_started":1}
I20260808 04:08:23.694006 30512 raft_consensus.cc:482] T 63ba482b902c4de88778f48a2dc007fc P b4fff30c1f1d4b2ab709870b18a9f6e2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:08:23.694285 30512 raft_consensus.cc:504] T 63ba482b902c4de88778f48a2dc007fc P b4fff30c1f1d4b2ab709870b18a9f6e2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87132debb51140cb9d97a17d217b6852" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 42143 } } peers { permanent_uuid: "b4fff30c1f1d4b2ab709870b18a9f6e2" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 34443 } } peers { permanent_uuid: "f3a563398cf94de0ae7d1c748d99faab" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 33807 } }
I20260808 04:08:23.696095 30512 leader_election.cc:288] T 63ba482b902c4de88778f48a2dc007fc P b4fff30c1f1d4b2ab709870b18a9f6e2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 87132debb51140cb9d97a17d217b6852 (127.5.83.3:42143), f3a563398cf94de0ae7d1c748d99faab (127.5.83.2:33807)
Aug 08 04:08:23 dist-test-slave-1l3l krb5kdc[28123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162102, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.1@KRBTEST.COM for kudu/127.5.83.3@KRBTEST.COM
Aug 08 04:08:23 dist-test-slave-1l3l krb5kdc[28123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162102, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.1@KRBTEST.COM for kudu/127.5.83.2@KRBTEST.COM
I20260808 04:08:23.717865 30509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:23.696346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.3:42143 (local address 127.5.83.1:39047)
0808 04:08:23.696598 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:23.696636 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0808 04:08:23.697202 (+ 566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:08:23.701534 (+ 4332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:08:23.701541 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:08:23.702449 (+ 908us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:08:23.703519 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:23.703546 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:23.705871 (+ 2325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:23.705877 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:23.707316 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:23.707328 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:23.708335 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:23.708343 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:23.708572 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:23.709299 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:08:23.709316 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:08:23.711503 (+ 2187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:08:23.714636 (+ 3133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:08:23.714645 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:08:23.714659 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:08:23.715051 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:08:23.715478 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:08:23.715483 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:08:23.715487 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:08:23.715658 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:08:23.716429 (+ 771us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:08:23.716437 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:08:23.716756 (+ 319us) client_negotiation.cc:769] Sending connection context
0808 04:08:23.717146 (+ 390us) client_negotiation.cc:240] Negotiation successful
0808 04:08:23.717562 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":116,"spinlock_wait_cycles":23552}
I20260808 04:08:23.718279 30517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:23.696709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.1:57795 (local address 127.5.83.2:33807)
0808 04:08:23.698571 (+ 1862us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:23.698585 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:23.698620 (+ 35us) server_negotiation.cc:407] Connection header received
0808 04:08:23.700654 (+ 2034us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:23.700663 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:23.701100 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:23.701456 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:08:23.703753 (+ 2297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:23.705647 (+ 1894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:23.707454 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:23.708094 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:23.708254 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:23.712152 (+ 3898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:08:23.712159 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:08:23.712170 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:08:23.712202 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:08:23.714955 (+ 2753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:08:23.715576 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:08:23.715582 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:08:23.715586 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:08:23.715782 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:08:23.716142 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:08:23.716147 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:08:23.716151 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:08:23.716703 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:08:23.716858 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:23.717588 (+ 730us) server_negotiation.cc:299] Negotiation successful
0808 04:08:23.717902 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1679,"thread_start_us":1558,"threads_started":1}
I20260808 04:08:23.718977 30514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:23.696583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.1:39047 (local address 127.5.83.3:42143)
0808 04:08:23.698276 (+ 1693us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:23.698291 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:23.698340 (+ 49us) server_negotiation.cc:407] Connection header received
0808 04:08:23.700590 (+ 2250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:23.700602 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:23.701101 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:23.701456 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:08:23.703754 (+ 2298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:23.705651 (+ 1897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:23.707448 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:23.708102 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:23.708262 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:23.711806 (+ 3544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:08:23.711814 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:08:23.711826 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:08:23.711860 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:08:23.714498 (+ 2638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:08:23.715176 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:08:23.715184 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:08:23.715190 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:08:23.715358 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:08:23.715787 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:08:23.715791 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:08:23.715795 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:08:23.716277 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:08:23.716440 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:23.718291 (+ 1851us) server_negotiation.cc:299] Negotiation successful
0808 04:08:23.718675 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1523,"thread_start_us":1402,"threads_started":1}
I20260808 04:08:23.718993 30513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:23.696366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.2:33807 (local address 127.5.83.1:57795)
0808 04:08:23.697968 (+ 1602us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:23.697986 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 04:08:23.698093 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:08:23.701730 (+ 3637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:08:23.701735 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:08:23.702449 (+ 714us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:08:23.703519 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:23.703546 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:23.705872 (+ 2326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:23.705876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:23.707319 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:23.707332 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:23.708288 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:23.708294 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:23.708570 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:23.709252 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:08:23.709278 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:08:23.711926 (+ 2648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:08:23.715086 (+ 3160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:08:23.715091 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:08:23.715095 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:08:23.715453 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:08:23.715893 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:08:23.715898 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:08:23.715901 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:08:23.716042 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:08:23.716833 (+ 791us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:08:23.716838 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:08:23.716971 (+ 133us) client_negotiation.cc:769] Sending connection context
0808 04:08:23.717137 (+ 166us) client_negotiation.cc:240] Negotiation successful
0808 04:08:23.718725 (+ 1588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1456,"mutex_wait_us":354,"spinlock_wait_cycles":33920,"thread_start_us":1085,"threads_started":1}
W20260808 04:08:23.721740 29945 leader_election.cc:341] T 63ba482b902c4de88778f48a2dc007fc P b4fff30c1f1d4b2ab709870b18a9f6e2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f3a563398cf94de0ae7d1c748d99faab (127.5.83.2:33807): Not found: Tablet not found: 63ba482b902c4de88778f48a2dc007fc
W20260808 04:08:23.722038 29946 leader_election.cc:341] T 63ba482b902c4de88778f48a2dc007fc P b4fff30c1f1d4b2ab709870b18a9f6e2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 87132debb51140cb9d97a17d217b6852 (127.5.83.3:42143): Not found: Tablet not found: 63ba482b902c4de88778f48a2dc007fc
I20260808 04:08:23.722191 29946 leader_election.cc:302] T 63ba482b902c4de88778f48a2dc007fc P b4fff30c1f1d4b2ab709870b18a9f6e2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b4fff30c1f1d4b2ab709870b18a9f6e2; no voters: 87132debb51140cb9d97a17d217b6852, f3a563398cf94de0ae7d1c748d99faab
I20260808 04:08:23.722579 30512 raft_consensus.cc:2726] T 63ba482b902c4de88778f48a2dc007fc P b4fff30c1f1d4b2ab709870b18a9f6e2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 04:08:23.810587 30519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:23.809772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:38749 (local address 127.5.83.2:38027)
0808 04:08:23.810223 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:23.810355 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:38749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":214,"threads_started":1}
I20260808 04:08:24.251224 30520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:24.250483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:38749 (local address 127.5.83.3:51991)
0808 04:08:24.250920 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:24.251021 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:38749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":196,"threads_started":1}
I20260808 04:08:24.382412 30521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:24.381788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:38749 (local address 127.5.83.1:50443)
0808 04:08:24.382188 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:24.382265 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:38749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":206,"threads_started":1}
I20260808 04:08:24.812100 30522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:24.811357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:38749 (local address 127.5.83.2:37023)
0808 04:08:24.811863 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:24.811947 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:38749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":250,"threads_started":1}
I20260808 04:08:25.252848 30524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:25.252103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:38749 (local address 127.5.83.3:56661)
0808 04:08:25.252572 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:25.252660 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:38749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":256,"threads_started":1}
I20260808 04:08:25.383986 30525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:25.383171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:38749 (local address 127.5.83.1:58251)
0808 04:08:25.383674 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:25.383804 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:38749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":261,"threads_started":1}
I20260808 04:08:25.702553 30527 raft_consensus.cc:482] T 63ba482b902c4de88778f48a2dc007fc P b4fff30c1f1d4b2ab709870b18a9f6e2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:08:25.702792 30527 raft_consensus.cc:504] T 63ba482b902c4de88778f48a2dc007fc P b4fff30c1f1d4b2ab709870b18a9f6e2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87132debb51140cb9d97a17d217b6852" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 42143 } } peers { permanent_uuid: "b4fff30c1f1d4b2ab709870b18a9f6e2" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 34443 } } peers { permanent_uuid: "f3a563398cf94de0ae7d1c748d99faab" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 33807 } }
I20260808 04:08:25.703430 30527 leader_election.cc:288] T 63ba482b902c4de88778f48a2dc007fc P b4fff30c1f1d4b2ab709870b18a9f6e2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 87132debb51140cb9d97a17d217b6852 (127.5.83.3:42143), f3a563398cf94de0ae7d1c748d99faab (127.5.83.2:33807)
W20260808 04:08:25.704416 29945 leader_election.cc:341] T 63ba482b902c4de88778f48a2dc007fc P b4fff30c1f1d4b2ab709870b18a9f6e2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f3a563398cf94de0ae7d1c748d99faab (127.5.83.2:33807): Not found: Tablet not found: 63ba482b902c4de88778f48a2dc007fc
W20260808 04:08:25.704595 29946 leader_election.cc:341] T 63ba482b902c4de88778f48a2dc007fc P b4fff30c1f1d4b2ab709870b18a9f6e2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 87132debb51140cb9d97a17d217b6852 (127.5.83.3:42143): Not found: Tablet not found: 63ba482b902c4de88778f48a2dc007fc
I20260808 04:08:25.704722 29946 leader_election.cc:302] T 63ba482b902c4de88778f48a2dc007fc P b4fff30c1f1d4b2ab709870b18a9f6e2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b4fff30c1f1d4b2ab709870b18a9f6e2; no voters: 87132debb51140cb9d97a17d217b6852, f3a563398cf94de0ae7d1c748d99faab
I20260808 04:08:25.704985 30527 raft_consensus.cc:2726] T 63ba482b902c4de88778f48a2dc007fc P b4fff30c1f1d4b2ab709870b18a9f6e2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 04:08:25.813639 30528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:25.812940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:38749 (local address 127.5.83.2:43587)
0808 04:08:25.813407 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:25.813484 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:38749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":246,"threads_started":1}
I20260808 04:08:26.254308 30530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:26.253592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:38749 (local address 127.5.83.3:52563)
0808 04:08:26.254064 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:26.254143 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:38749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":239,"threads_started":1}
I20260808 04:08:26.385444 30531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:26.384738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:38749 (local address 127.5.83.1:59445)
0808 04:08:26.385191 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:26.385267 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:38749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":248,"threads_started":1}
I20260808 04:08:26.815199 30532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:26.814472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:38749 (local address 127.5.83.2:47431)
0808 04:08:26.814933 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:26.815011 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:38749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":232,"threads_started":1}
I20260808 04:08:27.255919 30533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:27.255140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:38749 (local address 127.5.83.3:58863)
0808 04:08:27.255600 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:27.255677 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:38749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":223,"threads_started":1}
I20260808 04:08:27.387118 30534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:27.386373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:38749 (local address 127.5.83.1:51807)
0808 04:08:27.386863 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:27.386944 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:38749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":211,"threads_started":1}
I20260808 04:08:27.816715 30535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:27.816039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:38749 (local address 127.5.83.2:38047)
0808 04:08:27.816482 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:27.816558 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:38749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":235,"threads_started":1}
I20260808 04:08:28.257427 30536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:28.256729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:38749 (local address 127.5.83.3:58967)
0808 04:08:28.257187 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:28.257269 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:38749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":255,"threads_started":1}
I20260808 04:08:28.388631 30537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:28.387906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:38749 (local address 127.5.83.1:48609)
0808 04:08:28.388375 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:28.388475 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:38749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":237,"threads_started":1}
I20260808 04:08:28.471500 30538 raft_consensus.cc:482] T 63ba482b902c4de88778f48a2dc007fc P b4fff30c1f1d4b2ab709870b18a9f6e2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:08:28.471724 30538 raft_consensus.cc:504] T 63ba482b902c4de88778f48a2dc007fc P b4fff30c1f1d4b2ab709870b18a9f6e2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87132debb51140cb9d97a17d217b6852" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 42143 } } peers { permanent_uuid: "b4fff30c1f1d4b2ab709870b18a9f6e2" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 34443 } } peers { permanent_uuid: "f3a563398cf94de0ae7d1c748d99faab" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 33807 } }
I20260808 04:08:28.472360 30538 leader_election.cc:288] T 63ba482b902c4de88778f48a2dc007fc P b4fff30c1f1d4b2ab709870b18a9f6e2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 87132debb51140cb9d97a17d217b6852 (127.5.83.3:42143), f3a563398cf94de0ae7d1c748d99faab (127.5.83.2:33807)
W20260808 04:08:28.473240 29945 leader_election.cc:341] T 63ba482b902c4de88778f48a2dc007fc P b4fff30c1f1d4b2ab709870b18a9f6e2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f3a563398cf94de0ae7d1c748d99faab (127.5.83.2:33807): Not found: Tablet not found: 63ba482b902c4de88778f48a2dc007fc
W20260808 04:08:28.473426 29946 leader_election.cc:341] T 63ba482b902c4de88778f48a2dc007fc P b4fff30c1f1d4b2ab709870b18a9f6e2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 87132debb51140cb9d97a17d217b6852 (127.5.83.3:42143): Not found: Tablet not found: 63ba482b902c4de88778f48a2dc007fc
I20260808 04:08:28.473523 29946 leader_election.cc:302] T 63ba482b902c4de88778f48a2dc007fc P b4fff30c1f1d4b2ab709870b18a9f6e2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b4fff30c1f1d4b2ab709870b18a9f6e2; no voters: 87132debb51140cb9d97a17d217b6852, f3a563398cf94de0ae7d1c748d99faab
I20260808 04:08:28.473726 30538 raft_consensus.cc:2726] T 63ba482b902c4de88778f48a2dc007fc P b4fff30c1f1d4b2ab709870b18a9f6e2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 04:08:28.818187 30540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:28.817496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:38749 (local address 127.5.83.2:46131)
0808 04:08:28.817944 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:28.818020 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:38749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":232,"threads_started":1}
I20260808 04:08:29.259018 30542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:29.258230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:38749 (local address 127.5.83.3:49867)
0808 04:08:29.258706 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:29.258788 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:38749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":255,"threads_started":1}
I20260808 04:08:29.390187 30543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:29.389391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:38749 (local address 127.5.83.1:54937)
0808 04:08:29.389877 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:29.389987 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:38749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":266,"threads_started":1}
I20260808 04:08:29.819772 30544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:29.818952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:38749 (local address 127.5.83.2:32795)
0808 04:08:29.819466 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:29.819551 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:38749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":233,"threads_started":1}
I20260808 04:08:30.260807 30545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:30.260000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:38749 (local address 127.5.83.3:41941)
0808 04:08:30.260494 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:30.260592 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:38749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":225,"threads_started":1}
I20260808 04:08:30.391657 30546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:30.390961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:38749 (local address 127.5.83.1:40255)
0808 04:08:30.391419 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:30.391505 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:38749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":235,"threads_started":1}
I20260808 04:08:30.821352 30547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:30.820645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:38749 (local address 127.5.83.2:50267)
0808 04:08:30.821072 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:30.821158 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:38749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":227,"threads_started":1}
I20260808 04:08:31.262488 30549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:31.261751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:38749 (local address 127.5.83.3:59095)
0808 04:08:31.262187 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:31.262287 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:38749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":237,"threads_started":1}
I20260808 04:08:31.393177 30550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:31.392465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:38749 (local address 127.5.83.1:37393)
0808 04:08:31.392905 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:31.393005 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:38749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":228,"threads_started":1}
I20260808 04:08:31.822911 30551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:31.822235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:38749 (local address 127.5.83.2:53955)
0808 04:08:31.822674 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:31.822755 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:38749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":215,"threads_started":1}
I20260808 04:08:32.264155 30553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:32.263424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:38749 (local address 127.5.83.3:56251)
0808 04:08:32.263891 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:32.263984 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:38749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":211,"threads_started":1}
I20260808 04:08:32.394713 30555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:32.394029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:38749 (local address 127.5.83.1:34277)
0808 04:08:32.394485 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:32.394557 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:38749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":222,"threads_started":1}
I20260808 04:08:32.824523 30559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:32.823817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:38749 (local address 127.5.83.2:44263)
0808 04:08:32.824263 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:32.824343 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:38749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":235,"threads_started":1}
I20260808 04:08:33.090291 30561 raft_consensus.cc:482] T 63ba482b902c4de88778f48a2dc007fc P b4fff30c1f1d4b2ab709870b18a9f6e2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:08:33.090517 30561 raft_consensus.cc:504] T 63ba482b902c4de88778f48a2dc007fc P b4fff30c1f1d4b2ab709870b18a9f6e2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87132debb51140cb9d97a17d217b6852" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 42143 } } peers { permanent_uuid: "b4fff30c1f1d4b2ab709870b18a9f6e2" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 34443 } } peers { permanent_uuid: "f3a563398cf94de0ae7d1c748d99faab" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 33807 } }
I20260808 04:08:33.091151 30561 leader_election.cc:288] T 63ba482b902c4de88778f48a2dc007fc P b4fff30c1f1d4b2ab709870b18a9f6e2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 87132debb51140cb9d97a17d217b6852 (127.5.83.3:42143), f3a563398cf94de0ae7d1c748d99faab (127.5.83.2:33807)
W20260808 04:08:33.092203 29945 leader_election.cc:341] T 63ba482b902c4de88778f48a2dc007fc P b4fff30c1f1d4b2ab709870b18a9f6e2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f3a563398cf94de0ae7d1c748d99faab (127.5.83.2:33807): Not found: Tablet not found: 63ba482b902c4de88778f48a2dc007fc
W20260808 04:08:33.092497 29946 leader_election.cc:341] T 63ba482b902c4de88778f48a2dc007fc P b4fff30c1f1d4b2ab709870b18a9f6e2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 87132debb51140cb9d97a17d217b6852 (127.5.83.3:42143): Not found: Tablet not found: 63ba482b902c4de88778f48a2dc007fc
I20260808 04:08:33.092648 29946 leader_election.cc:302] T 63ba482b902c4de88778f48a2dc007fc P b4fff30c1f1d4b2ab709870b18a9f6e2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b4fff30c1f1d4b2ab709870b18a9f6e2; no voters: 87132debb51140cb9d97a17d217b6852, f3a563398cf94de0ae7d1c748d99faab
I20260808 04:08:33.092900 30561 raft_consensus.cc:2726] T 63ba482b902c4de88778f48a2dc007fc P b4fff30c1f1d4b2ab709870b18a9f6e2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 04:08:33.265854 30562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:33.264992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:38749 (local address 127.5.83.3:55209)
0808 04:08:33.265499 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:33.265598 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:38749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":297,"threads_started":1}
I20260808 04:08:33.396412 30563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:33.395581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:38749 (local address 127.5.83.1:35159)
0808 04:08:33.396116 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:33.396197 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:38749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":282,"threads_started":1}
I20260808 04:08:33.826014 30566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:33.825310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:38749 (local address 127.5.83.2:37213)
0808 04:08:33.825763 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:33.825843 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:38749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":232,"threads_started":1}
I20260808 04:08:34.267412 30567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:34.266716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:38749 (local address 127.5.83.3:56979)
0808 04:08:34.267175 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:34.267254 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:38749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":224,"threads_started":1}
I20260808 04:08:34.397868 30568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:34.397198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:38749 (local address 127.5.83.1:59355)
0808 04:08:34.397623 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:34.397700 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:38749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":223,"threads_started":1}
I20260808 04:08:34.827504 30569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:34.826798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:38749 (local address 127.5.83.2:58105)
0808 04:08:34.827241 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:34.827321 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:38749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":234,"threads_started":1}
I20260808 04:08:35.268965 30570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:35.268215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:38749 (local address 127.5.83.3:33107)
0808 04:08:35.268676 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:35.268755 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:38749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":257,"threads_started":1}
I20260808 04:08:35.399377 30571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:35.398683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:38749 (local address 127.5.83.1:51095)
0808 04:08:35.399119 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:35.399201 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:38749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":228,"threads_started":1}
I20260808 04:08:35.829036 30573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:35.828348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:38749 (local address 127.5.83.2:48307)
0808 04:08:35.828784 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:35.828859 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:38749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":220,"threads_started":1}
I20260808 04:08:36.270562 30574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:36.269777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:38749 (local address 127.5.83.3:43009)
0808 04:08:36.270295 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:36.270403 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:38749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":292,"threads_started":1}
I20260808 04:08:36.400851 30575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:36.400162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:38749 (local address 127.5.83.1:36357)
0808 04:08:36.400574 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:36.400647 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:38749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":237,"threads_started":1}
I20260808 04:08:36.830605 30576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:36.829868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:38749 (local address 127.5.83.2:40079)
0808 04:08:36.830324 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:36.830409 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:38749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":237,"threads_started":1}
I20260808 04:08:37.272146 30580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:37.271354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:38749 (local address 127.5.83.3:58157)
0808 04:08:37.271848 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:37.271944 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:38749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":231,"threads_started":1}
I20260808 04:08:37.402653 30581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:37.401966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:38749 (local address 127.5.83.1:33559)
0808 04:08:37.402402 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:37.402485 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:38749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":202,"threads_started":1}
I20260808 04:08:37.832295 30582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:37.831596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:38749 (local address 127.5.83.2:39675)
0808 04:08:37.832048 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:37.832141 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:38749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":246,"threads_started":1}
I20260808 04:08:38.253021 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 29862
I20260808 04:08:38.267665 30583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:38.254090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43760 (local address 127.5.83.1:34443)
0808 04:08:38.254661 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:38.254671 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:38.254712 (+ 41us) server_negotiation.cc:407] Connection header received
0808 04:08:38.254801 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:38.254810 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:38.255094 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:38.255270 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:08:38.256875 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:38.257964 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:38.259230 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:38.259844 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:38.259942 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:38.262505 (+ 2563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:08:38.262521 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:08:38.262534 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:08:38.262568 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:08:38.265246 (+ 2678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:08:38.265857 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:08:38.265863 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:08:38.265866 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:08:38.265997 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:08:38.266298 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:08:38.266301 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:08:38.266303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:08:38.266653 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:08:38.266887 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:38.267108 (+ 221us) server_negotiation.cc:299] Negotiation successful
0808 04:08:38.267299 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":256,"threads_started":1}
I20260808 04:08:38.268989 30100 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:08:38.273867 30585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:38.273161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:38749 (local address 127.5.83.3:40163)
0808 04:08:38.273618 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:38.273702 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.62:38749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":228,"threads_started":1}
I20260808 04:08:38.359505 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 29862
I20260808 04:08:38.383467 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 30114
I20260808 04:08:38.396584 30586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:38.384242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35248 (local address 127.5.83.2:33807)
0808 04:08:38.384672 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:38.384677 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:38.384695 (+ 18us) server_negotiation.cc:407] Connection header received
0808 04:08:38.384751 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:38.384755 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:38.384814 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:38.384903 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:08:38.386262 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:38.387088 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:38.388609 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:38.389027 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:38.389131 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:38.391650 (+ 2519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:08:38.391655 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:08:38.391658 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:08:38.391691 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:08:38.394259 (+ 2568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:08:38.394850 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:08:38.394854 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:08:38.394857 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:08:38.394980 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:08:38.395249 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:08:38.395251 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:08:38.395253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:08:38.395538 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:08:38.395769 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:38.396031 (+ 262us) server_negotiation.cc:299] Negotiation successful
0808 04:08:38.396195 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":226,"threads_started":1}
I20260808 04:08:38.397203 30296 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:08:38.479573 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 30114
I20260808 04:08:38.501119 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 30309
I20260808 04:08:38.514478 30588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:38.501951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36868 (local address 127.5.83.3:42143)
0808 04:08:38.502558 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:38.502563 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:38.502582 (+ 19us) server_negotiation.cc:407] Connection header received
0808 04:08:38.502640 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:38.502644 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:38.502713 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:38.502799 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:08:38.504232 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:38.505035 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:38.506242 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:38.506675 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:38.506774 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:38.509415 (+ 2641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:08:38.509420 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:08:38.509424 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:08:38.509457 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:08:38.512167 (+ 2710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:08:38.512825 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:08:38.512829 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:08:38.512832 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:08:38.512977 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:08:38.513263 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:08:38.513265 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:08:38.513267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:08:38.513527 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:08:38.513726 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:38.513928 (+ 202us) server_negotiation.cc:299] Negotiation successful
0808 04:08:38.514092 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":299,"threads_started":1}
I20260808 04:08:38.515159 30497 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:08:38.599651 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 30309
2026-08-08T04:08:38Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64384 ms)
[----------] 30 tests from SecurityITest (291939 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:08:38 dist-test-slave-1l3l krb5kdc[30594](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:08:38 dist-test-slave-1l3l krb5kdc[30594](info): set up 2 sockets
Aug 08 04:08:38 dist-test-slave-1l3l krb5kdc[30594](info): commencing operation
krb5kdc: starting...
W20260808 04:08:39.803280 5452 mini_kdc.cc:121] Time spent starting KDC: real 1.156s user 0.002s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/krb5kdc/test-user.keytab.
Aug 08 04:08:39 dist-test-slave-1l3l krb5kdc[30594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:08:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:08:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.83.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab.
Entry for principal kudu/127.5.83.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab.
I20260808 04:08:40.056587 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:46615
--webserver_interface=127.5.83.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab
--principal=kudu/127.5.83.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:38617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.62:46615
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:08:40.386348 30611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:08:40.386641 30611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:08:40.386778 30611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:08:40.393805 30611 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:08:40.393918 30611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:08:40.394003 30611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:08:40.394078 30611 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:08:40.394140 30611 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:08:40.403533 30611 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:38617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.62:46615
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab
--principal=kudu/127.5.83.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:46615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:08:40.404781 30611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:08:40.406095 30611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:08:40.417438 30622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:08:40.417465 30620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:08:40.418080 30611 server_base.cc:1034] running on GCE node
W20260808 04:08:40.418241 30619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:08:40.418982 30611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:08:40.420336 30611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:08:40.421535 30611 hybrid_clock.cc:648] HybridClock initialized: now 1786162120421487 us; error 42 us; skew 500 ppm
I20260808 04:08:40.421833 30611 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:08:40 dist-test-slave-1l3l krb5kdc[30594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162120, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:08:40.424803 30611 init.cc:377] Logged in from keytab as kudu/127.5.83.62@KRBTEST.COM (short username kudu)
I20260808 04:08:40.426142 30611 webserver.cc:460] Webserver started at http://127.5.83.62:40389/ using document root <none> and password file <none>
I20260808 04:08:40.426759 30611 fs_manager.cc:359] Metadata directory not provided
I20260808 04:08:40.426873 30611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:08:40.427147 30611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:08:40.430043 30611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/minicluster-data/master-0/data/instance:
uuid: "6f9c51b72c6e4f0a8d53b56e8559674a"
format_stamp: "Formatted at 2026-08-08 04:08:40 on dist-test-slave-1l3l"
server_key: "5fed6b79ac1b594896a63894ae6d09cf"
server_key_iv: "9699fe00ef4a39f247df576f8e816d1d"
server_key_version: "encryptionkey@0"
I20260808 04:08:40.430783 30611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/minicluster-data/master-0/wal/instance:
uuid: "6f9c51b72c6e4f0a8d53b56e8559674a"
format_stamp: "Formatted at 2026-08-08 04:08:40 on dist-test-slave-1l3l"
server_key: "5fed6b79ac1b594896a63894ae6d09cf"
server_key_iv: "9699fe00ef4a39f247df576f8e816d1d"
server_key_version: "encryptionkey@0"
I20260808 04:08:40.435664 30611 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260808 04:08:40.438867 30631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:08:40.440130 30611 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:08:40.440289 30611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/minicluster-data/master-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/minicluster-data/master-0/wal
uuid: "6f9c51b72c6e4f0a8d53b56e8559674a"
format_stamp: "Formatted at 2026-08-08 04:08:40 on dist-test-slave-1l3l"
server_key: "5fed6b79ac1b594896a63894ae6d09cf"
server_key_iv: "9699fe00ef4a39f247df576f8e816d1d"
server_key_version: "encryptionkey@0"
I20260808 04:08:40.440469 30611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:08:40.455041 30611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:08:40.458990 30611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:08:40.459239 30611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:08:40.476361 30611 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.62:46615
I20260808 04:08:40.476454 30712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.62:46615 every 8 connection(s)
I20260808 04:08:40.477618 30611 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
I20260808 04:08:40.480543 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 30611
I20260808 04:08:40.480792 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/minicluster-data/master-0/wal/instance
I20260808 04:08:40.481491 30713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:08:40.481652 5452 external_mini_cluster.cc:1428] Setting key 75c7415386317362bc8c12be844723e5
I20260808 04:08:40.495267 30713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f9c51b72c6e4f0a8d53b56e8559674a: Bootstrap starting.
Aug 08 04:08:40 dist-test-slave-1l3l krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:08:40.498716 30713 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f9c51b72c6e4f0a8d53b56e8559674a: Neither blocks nor log segments found. Creating new log.
I20260808 04:08:40.499867 30713 log.cc:824] T 00000000000000000000000000000000 P 6f9c51b72c6e4f0a8d53b56e8559674a: Log is configured to *not* fsync() on all Append() calls
I20260808 04:08:40.502745 30713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f9c51b72c6e4f0a8d53b56e8559674a: No bootstrap required, opened a new log
I20260808 04:08:40.504338 30718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:40.483594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34894 (local address 127.5.83.62:46615)
0808 04:08:40.484382 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:40.484396 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:40.484442 (+ 46us) server_negotiation.cc:407] Connection header received
0808 04:08:40.486096 (+ 1654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:40.486115 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:40.486452 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:40.486742 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:08:40.488662 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:40.489870 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:40.491670 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:40.492387 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:40.492516 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:40.497482 (+ 4966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:08:40.497489 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:08:40.497500 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:08:40.497536 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:08:40.500542 (+ 3006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:08:40.501211 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:08:40.501217 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:08:40.501222 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:08:40.501374 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:08:40.501778 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:08:40.501781 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:08:40.501783 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:08:40.502318 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:08:40.502518 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:40.503023 (+ 505us) server_negotiation.cc:299] Negotiation successful
0808 04:08:40.503340 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":377,"threads_started":1}
I20260808 04:08:40.510142 30713 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f9c51b72c6e4f0a8d53b56e8559674a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f9c51b72c6e4f0a8d53b56e8559674a" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 46615 } }
I20260808 04:08:40.510465 30713 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f9c51b72c6e4f0a8d53b56e8559674a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:08:40.510560 30713 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f9c51b72c6e4f0a8d53b56e8559674a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f9c51b72c6e4f0a8d53b56e8559674a, State: Initialized, Role: FOLLOWER
I20260808 04:08:40.511053 30713 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f9c51b72c6e4f0a8d53b56e8559674a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f9c51b72c6e4f0a8d53b56e8559674a" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 46615 } }
I20260808 04:08:40.511263 30713 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f9c51b72c6e4f0a8d53b56e8559674a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:08:40.511392 30713 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f9c51b72c6e4f0a8d53b56e8559674a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:08:40.511554 30713 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f9c51b72c6e4f0a8d53b56e8559674a [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:08:40.513669 30713 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f9c51b72c6e4f0a8d53b56e8559674a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f9c51b72c6e4f0a8d53b56e8559674a" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 46615 } }
I20260808 04:08:40.514092 30713 leader_election.cc:302] T 00000000000000000000000000000000 P 6f9c51b72c6e4f0a8d53b56e8559674a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f9c51b72c6e4f0a8d53b56e8559674a; no voters:
I20260808 04:08:40.514575 30713 leader_election.cc:288] T 00000000000000000000000000000000 P 6f9c51b72c6e4f0a8d53b56e8559674a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:08:40.514689 30720 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f9c51b72c6e4f0a8d53b56e8559674a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:08:40.515399 30720 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f9c51b72c6e4f0a8d53b56e8559674a [term 1 LEADER]: Becoming Leader. State: Replica: 6f9c51b72c6e4f0a8d53b56e8559674a, State: Running, Role: LEADER
I20260808 04:08:40.515962 30720 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f9c51b72c6e4f0a8d53b56e8559674a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f9c51b72c6e4f0a8d53b56e8559674a" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 46615 } }
I20260808 04:08:40.516606 30713 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f9c51b72c6e4f0a8d53b56e8559674a [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:08:40.518697 30722 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f9c51b72c6e4f0a8d53b56e8559674a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f9c51b72c6e4f0a8d53b56e8559674a. Latest consensus state: current_term: 1 leader_uuid: "6f9c51b72c6e4f0a8d53b56e8559674a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f9c51b72c6e4f0a8d53b56e8559674a" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 46615 } } }
I20260808 04:08:40.518884 30721 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f9c51b72c6e4f0a8d53b56e8559674a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f9c51b72c6e4f0a8d53b56e8559674a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f9c51b72c6e4f0a8d53b56e8559674a" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 46615 } } }
I20260808 04:08:40.519146 30721 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f9c51b72c6e4f0a8d53b56e8559674a [sys.catalog]: This master's current role is: LEADER
I20260808 04:08:40.519155 30722 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f9c51b72c6e4f0a8d53b56e8559674a [sys.catalog]: This master's current role is: LEADER
I20260808 04:08:40.520041 30728 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:08:40.525496 30728 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:08:40.533878 30728 catalog_manager.cc:1347] Generated new cluster ID: ff64c71e187143e99a7f0c8403985bdc
I20260808 04:08:40.533986 30728 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:08:40.547029 30728 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:08:40.548823 30728 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:08:40.556782 30728 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f9c51b72c6e4f0a8d53b56e8559674a: Generated new TSK 0
I20260808 04:08:40.557678 30728 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.83.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.1.keytab.
Entry for principal kudu/127.5.83.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.1.keytab.
I20260808 04:08:40.618834 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.1:0
--local_ip_for_outbound_sockets=127.5.83.1
--webserver_interface=127.5.83.1
--webserver_port=0
--tserver_master_addrs=127.5.83.62:46615
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.1.keytab
--principal=kudu/127.5.83.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:38617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:08:40.943025 30750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:08:40.943377 30750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:08:40.943519 30750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:08:40.950762 30750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:08:40.950879 30750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:08:40.951035 30750 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.1
I20260808 04:08:40.960613 30750 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:38617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.1.keytab
--principal=kudu/127.5.83.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.83.1
--webserver_port=0
--tserver_master_addrs=127.5.83.62:46615
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.1
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:08:40.961838 30750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:08:40.963078 30750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:08:40.975803 30757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:08:40.975876 30760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:08:40.975847 30758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:08:40.976861 30750 server_base.cc:1034] running on GCE node
I20260808 04:08:40.977423 30750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:08:40.978286 30750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:08:40.979485 30750 hybrid_clock.cc:648] HybridClock initialized: now 1786162120979456 us; error 58 us; skew 500 ppm
I20260808 04:08:40.979822 30750 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:08:40 dist-test-slave-1l3l krb5kdc[30594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162120, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:08:40.982847 30750 init.cc:377] Logged in from keytab as kudu/127.5.83.1@KRBTEST.COM (short username kudu)
I20260808 04:08:40.984256 30750 webserver.cc:460] Webserver started at http://127.5.83.1:34819/ using document root <none> and password file <none>
I20260808 04:08:40.984895 30750 fs_manager.cc:359] Metadata directory not provided
I20260808 04:08:40.984998 30750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:08:40.985248 30750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:08:40.988159 30750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/minicluster-data/ts-0/data/instance:
uuid: "ec04c83d4a824de9989627e8e51747f8"
format_stamp: "Formatted at 2026-08-08 04:08:40 on dist-test-slave-1l3l"
server_key: "94b4e97a6ac52dc63a7ac61697564702"
server_key_iv: "91ae1ace32e577e5274d58031cce6562"
server_key_version: "encryptionkey@0"
I20260808 04:08:40.988895 30750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/minicluster-data/ts-0/wal/instance:
uuid: "ec04c83d4a824de9989627e8e51747f8"
format_stamp: "Formatted at 2026-08-08 04:08:40 on dist-test-slave-1l3l"
server_key: "94b4e97a6ac52dc63a7ac61697564702"
server_key_iv: "91ae1ace32e577e5274d58031cce6562"
server_key_version: "encryptionkey@0"
I20260808 04:08:40.993769 30750 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 04:08:40.997076 30769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:08:40.998274 30750 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260808 04:08:40.998477 30750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/minicluster-data/ts-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/minicluster-data/ts-0/wal
uuid: "ec04c83d4a824de9989627e8e51747f8"
format_stamp: "Formatted at 2026-08-08 04:08:40 on dist-test-slave-1l3l"
server_key: "94b4e97a6ac52dc63a7ac61697564702"
server_key_iv: "91ae1ace32e577e5274d58031cce6562"
server_key_version: "encryptionkey@0"
I20260808 04:08:40.998668 30750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:08:41.018509 30750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:08:41.022590 30750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:08:41.022857 30750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:08:41.023921 30750 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:08:41.028435 30750 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:08:41.028532 30750 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:08:41.028676 30750 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:08:41.028764 30750 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260808 04:08:41.060144 30750 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.1:40165
I20260808 04:08:41.060240 30934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.1:40165 every 8 connection(s)
I20260808 04:08:41.061383 30750 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
I20260808 04:08:41.065625 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 30750
I20260808 04:08:41.065852 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/minicluster-data/ts-0/wal/instance
I20260808 04:08:41.066679 5452 external_mini_cluster.cc:1428] Setting key be9ec35040ef07ec1050ec3cbd7c6d28
Aug 08 04:08:41 dist-test-slave-1l3l krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162120, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.1@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:08:41.082372 30938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:41.063869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.1:60173 (local address 127.5.83.62:46615)
0808 04:08:41.064321 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:41.064326 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:41.065278 (+ 952us) server_negotiation.cc:407] Connection header received
0808 04:08:41.067701 (+ 2423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:41.067729 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:41.067830 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:41.067958 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:08:41.069661 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:41.070368 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:41.072448 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:41.072843 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:41.072954 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:41.076844 (+ 3890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:08:41.076849 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:08:41.076852 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:08:41.076884 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:08:41.079331 (+ 2447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:08:41.080073 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:08:41.080077 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:08:41.080080 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:08:41.080199 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:08:41.080616 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:08:41.080619 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:08:41.080621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:08:41.080927 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:08:41.081063 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:41.081764 (+ 701us) server_negotiation.cc:299] Negotiation successful
0808 04:08:41.082010 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":223,"threads_started":1}
I20260808 04:08:41.083287 30939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:41.064419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:46615 (local address 127.5.83.1:60173)
0808 04:08:41.065087 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:41.065142 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0808 04:08:41.067415 (+ 2273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:08:41.068143 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:08:41.068151 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:08:41.068721 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:08:41.069484 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:41.069509 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:41.070527 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:41.070533 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:41.072291 (+ 1758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:41.072306 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:41.073076 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:41.073083 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:41.073356 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:41.074033 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:08:41.074065 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:08:41.076637 (+ 2572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:08:41.079468 (+ 2831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:08:41.079477 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:08:41.079493 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:08:41.079952 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:08:41.080303 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:08:41.080309 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:08:41.080312 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:08:41.080502 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:08:41.081052 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:08:41.081059 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:08:41.081375 (+ 316us) client_negotiation.cc:769] Sending connection context
0808 04:08:41.081625 (+ 250us) client_negotiation.cc:240] Negotiation successful
0808 04:08:41.082147 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":256,"threads_started":1}
I20260808 04:08:41.084790 30935 heartbeater.cc:347] Connected to a master server at 127.5.83.62:46615
I20260808 04:08:41.085079 30935 heartbeater.cc:464] Registering TS with master...
I20260808 04:08:41.085788 30935 heartbeater.cc:511] Master 127.5.83.62:46615 requested a full tablet report, sending...
I20260808 04:08:41.087996 30660 ts_manager.cc:194] Registered new tserver with Master: ec04c83d4a824de9989627e8e51747f8 (127.5.83.1:40165)
I20260808 04:08:41.089924 30660 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.1@KRBTEST.COM'} at 127.5.83.1:60173
WARNING: no policy specified for kudu/127.5.83.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.2.keytab.
Entry for principal kudu/127.5.83.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.2.keytab.
I20260808 04:08:41.128417 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.2:0
--local_ip_for_outbound_sockets=127.5.83.2
--webserver_interface=127.5.83.2
--webserver_port=0
--tserver_master_addrs=127.5.83.62:46615
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.2.keytab
--principal=kudu/127.5.83.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:38617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:08:41.459576 30944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:08:41.459920 30944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:08:41.460037 30944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:08:41.467082 30944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:08:41.467181 30944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:08:41.467298 30944 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.2
I20260808 04:08:41.476714 30944 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:38617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.2.keytab
--principal=kudu/127.5.83.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.83.2
--webserver_port=0
--tserver_master_addrs=127.5.83.62:46615
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.2
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:08:41.477845 30944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:08:41.479053 30944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:08:41.491787 30949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:08:41.491698 30952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:08:41.491763 30950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:08:41.492785 30944 server_base.cc:1034] running on GCE node
I20260808 04:08:41.493456 30944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:08:41.494275 30944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:08:41.495476 30944 hybrid_clock.cc:648] HybridClock initialized: now 1786162121495439 us; error 27 us; skew 500 ppm
I20260808 04:08:41.495798 30944 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:08:41 dist-test-slave-1l3l krb5kdc[30594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162121, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:08:41.498725 30944 init.cc:377] Logged in from keytab as kudu/127.5.83.2@KRBTEST.COM (short username kudu)
I20260808 04:08:41.500082 30944 webserver.cc:460] Webserver started at http://127.5.83.2:43869/ using document root <none> and password file <none>
I20260808 04:08:41.500746 30944 fs_manager.cc:359] Metadata directory not provided
I20260808 04:08:41.500862 30944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:08:41.501149 30944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:08:41.504105 30944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/minicluster-data/ts-1/data/instance:
uuid: "f5b3416abb7244e78131b28f7692aa33"
format_stamp: "Formatted at 2026-08-08 04:08:41 on dist-test-slave-1l3l"
server_key: "a683d9ada53babf91beadd81d2282ce3"
server_key_iv: "55ae66b25ca46437bca779921a8146fa"
server_key_version: "encryptionkey@0"
I20260808 04:08:41.504832 30944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/minicluster-data/ts-1/wal/instance:
uuid: "f5b3416abb7244e78131b28f7692aa33"
format_stamp: "Formatted at 2026-08-08 04:08:41 on dist-test-slave-1l3l"
server_key: "a683d9ada53babf91beadd81d2282ce3"
server_key_iv: "55ae66b25ca46437bca779921a8146fa"
server_key_version: "encryptionkey@0"
I20260808 04:08:41.509601 30944 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 04:08:41.512748 30961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:08:41.513981 30944 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:08:41.514129 30944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/minicluster-data/ts-1/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/minicluster-data/ts-1/wal
uuid: "f5b3416abb7244e78131b28f7692aa33"
format_stamp: "Formatted at 2026-08-08 04:08:41 on dist-test-slave-1l3l"
server_key: "a683d9ada53babf91beadd81d2282ce3"
server_key_iv: "55ae66b25ca46437bca779921a8146fa"
server_key_version: "encryptionkey@0"
I20260808 04:08:41.514297 30944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:08:41.533973 30944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:08:41.537555 30944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:08:41.537810 30944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:08:41.538666 30944 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:08:41.539907 30944 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:08:41.539980 30944 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:08:41.540095 30944 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:08:41.540146 30944 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:08:41.568645 30944 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.2:35969
I20260808 04:08:41.568692 31123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.2:35969 every 8 connection(s)
I20260808 04:08:41.569803 30944 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
I20260808 04:08:41.574738 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 30944
I20260808 04:08:41.574915 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/minicluster-data/ts-1/wal/instance
I20260808 04:08:41.575520 5452 external_mini_cluster.cc:1428] Setting key 8ca9f3878f1181d331c0f7abf80206c9
Aug 08 04:08:41 dist-test-slave-1l3l krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162121, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.2@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:08:41.588932 30938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:41.572187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.2:38387 (local address 127.5.83.62:46615)
0808 04:08:41.572349 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:41.572353 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:41.573471 (+ 1118us) server_negotiation.cc:407] Connection header received
0808 04:08:41.575667 (+ 2196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:41.575671 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:41.575792 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:41.575896 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:08:41.577545 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:41.578255 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:41.579751 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:41.580153 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:41.580267 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:41.583581 (+ 3314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:08:41.583586 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:08:41.583589 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:08:41.583619 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:08:41.586093 (+ 2474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:08:41.586747 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:08:41.586753 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:08:41.586756 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:08:41.586880 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:08:41.587294 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:08:41.587299 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:08:41.587301 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:08:41.587559 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:08:41.587684 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:41.588352 (+ 668us) server_negotiation.cc:299] Negotiation successful
0808 04:08:41.588580 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 04:08:41.589784 31126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:41.572618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:46615 (local address 127.5.83.2:38387)
0808 04:08:41.573314 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:41.573355 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 04:08:41.575363 (+ 2008us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:08:41.576177 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:08:41.576188 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:08:41.576783 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:08:41.577382 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:41.577407 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:41.578396 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:41.578402 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:41.579591 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:41.579603 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:41.580369 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:41.580376 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:41.580585 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:41.581180 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:08:41.581207 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:08:41.583349 (+ 2142us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:08:41.586219 (+ 2870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:08:41.586227 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:08:41.586242 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:08:41.586636 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:08:41.587001 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:08:41.587005 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:08:41.587010 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:08:41.587187 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:08:41.587691 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:08:41.587699 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:08:41.588012 (+ 313us) client_negotiation.cc:769] Sending connection context
0808 04:08:41.588239 (+ 227us) client_negotiation.cc:240] Negotiation successful
0808 04:08:41.588725 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":276,"threads_started":1}
I20260808 04:08:41.591121 31124 heartbeater.cc:347] Connected to a master server at 127.5.83.62:46615
I20260808 04:08:41.591380 31124 heartbeater.cc:464] Registering TS with master...
I20260808 04:08:41.592087 31124 heartbeater.cc:511] Master 127.5.83.62:46615 requested a full tablet report, sending...
I20260808 04:08:41.593441 30660 ts_manager.cc:194] Registered new tserver with Master: f5b3416abb7244e78131b28f7692aa33 (127.5.83.2:35969)
I20260808 04:08:41.594466 30660 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.2@KRBTEST.COM'} at 127.5.83.2:38387
WARNING: no policy specified for kudu/127.5.83.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.3.keytab.
Entry for principal kudu/127.5.83.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.3.keytab.
I20260808 04:08:41.635013 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.3:0
--local_ip_for_outbound_sockets=127.5.83.3
--webserver_interface=127.5.83.3
--webserver_port=0
--tserver_master_addrs=127.5.83.62:46615
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.3.keytab
--principal=kudu/127.5.83.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:38617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:08:41.959501 31131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:08:41.959790 31131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:08:41.959951 31131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:08:41.967092 31131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:08:41.967186 31131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:08:41.967332 31131 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.3
I20260808 04:08:41.977069 31131 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:38617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.3.keytab
--principal=kudu/127.5.83.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.83.3
--webserver_port=0
--tserver_master_addrs=127.5.83.62:46615
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.3
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:08:41.978228 31131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:08:41.979508 31131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:08:41.991688 31141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:08:41.991773 31139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:08:41.992134 31138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:08:41.992755 31131 server_base.cc:1034] running on GCE node
I20260808 04:08:41.993309 31131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:08:41.994140 31131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:08:41.995334 31131 hybrid_clock.cc:648] HybridClock initialized: now 1786162121995280 us; error 35 us; skew 500 ppm
I20260808 04:08:41.995611 31131 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:08:41 dist-test-slave-1l3l krb5kdc[30594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162121, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:08:41.998530 31131 init.cc:377] Logged in from keytab as kudu/127.5.83.3@KRBTEST.COM (short username kudu)
I20260808 04:08:41.999859 31131 webserver.cc:460] Webserver started at http://127.5.83.3:39765/ using document root <none> and password file <none>
I20260808 04:08:42.000478 31131 fs_manager.cc:359] Metadata directory not provided
I20260808 04:08:42.000584 31131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:08:42.000869 31131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:08:42.003734 31131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/minicluster-data/ts-2/data/instance:
uuid: "b547566264d54b3fb0bcd8465baac89c"
format_stamp: "Formatted at 2026-08-08 04:08:41 on dist-test-slave-1l3l"
server_key: "b698a97520c14fdea197e43a86a03301"
server_key_iv: "f966b9ed110e271b5b79f51e1dcd2d87"
server_key_version: "encryptionkey@0"
I20260808 04:08:42.004500 31131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/minicluster-data/ts-2/wal/instance:
uuid: "b547566264d54b3fb0bcd8465baac89c"
format_stamp: "Formatted at 2026-08-08 04:08:41 on dist-test-slave-1l3l"
server_key: "b698a97520c14fdea197e43a86a03301"
server_key_iv: "f966b9ed110e271b5b79f51e1dcd2d87"
server_key_version: "encryptionkey@0"
I20260808 04:08:42.009284 31131 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260808 04:08:42.012576 31153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:08:42.013770 31131 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:08:42.013912 31131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/minicluster-data/ts-2/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/minicluster-data/ts-2/wal
uuid: "b547566264d54b3fb0bcd8465baac89c"
format_stamp: "Formatted at 2026-08-08 04:08:41 on dist-test-slave-1l3l"
server_key: "b698a97520c14fdea197e43a86a03301"
server_key_iv: "f966b9ed110e271b5b79f51e1dcd2d87"
server_key_version: "encryptionkey@0"
I20260808 04:08:42.014086 31131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:08:42.052191 31131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:08:42.056023 31131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:08:42.056304 31131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:08:42.057331 31131 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:08:42.058640 31131 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:08:42.058714 31131 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:08:42.058825 31131 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:08:42.058876 31131 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:08:42.090396 31131 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.3:46773
I20260808 04:08:42.090426 31324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.3:46773 every 8 connection(s)
I20260808 04:08:42.091630 31131 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
I20260808 04:08:42.093014 30935 heartbeater.cc:503] Master 127.5.83.62:46615 was elected leader, sending a full tablet report...
I20260808 04:08:42.100965 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 31131
I20260808 04:08:42.101125 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/minicluster-data/ts-2/wal/instance
I20260808 04:08:42.101593 5452 external_mini_cluster.cc:1428] Setting key 9cb2835f0aeb65f48bbdce10ac8a192b
Aug 08 04:08:42 dist-test-slave-1l3l krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162121, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.3@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:08:42.110754 31329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:42.094115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.3:43083 (local address 127.5.83.62:46615)
0808 04:08:42.094586 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:42.094591 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:42.095488 (+ 897us) server_negotiation.cc:407] Connection header received
0808 04:08:42.097256 (+ 1768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:42.097260 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:42.097323 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:42.097421 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:08:42.098827 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:42.099598 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:42.101178 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:42.101604 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:42.101739 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:42.105456 (+ 3717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:08:42.105460 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:08:42.105464 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:08:42.105494 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:08:42.108173 (+ 2679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:08:42.108856 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:08:42.108860 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:08:42.108862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:08:42.108996 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:08:42.109374 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:08:42.109378 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:08:42.109380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:08:42.109646 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:08:42.109768 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:42.110334 (+ 566us) server_negotiation.cc:299] Negotiation successful
0808 04:08:42.110494 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":229,"threads_started":1}
I20260808 04:08:42.111572 31330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:42.094659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:46615 (local address 127.5.83.3:43083)
0808 04:08:42.095338 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:42.095381 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 04:08:42.097021 (+ 1640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:08:42.097590 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:08:42.097598 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:08:42.098068 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:08:42.098698 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:42.098715 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:42.099751 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:42.099756 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:42.101031 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:42.101054 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:42.101872 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:42.101882 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:42.102101 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:42.102824 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:08:42.102855 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:08:42.105261 (+ 2406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:08:42.108281 (+ 3020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:08:42.108287 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:08:42.108297 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:08:42.108759 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:08:42.109072 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:08:42.109075 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:08:42.109077 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:08:42.109241 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:08:42.109751 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:08:42.109757 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:08:42.110010 (+ 253us) client_negotiation.cc:769] Sending connection context
0808 04:08:42.110211 (+ 201us) client_negotiation.cc:240] Negotiation successful
0808 04:08:42.110549 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":270,"threads_started":1}
I20260808 04:08:42.112939 31326 heartbeater.cc:347] Connected to a master server at 127.5.83.62:46615
I20260808 04:08:42.113238 31326 heartbeater.cc:464] Registering TS with master...
I20260808 04:08:42.113906 31326 heartbeater.cc:511] Master 127.5.83.62:46615 requested a full tablet report, sending...
I20260808 04:08:42.115185 30660 ts_manager.cc:194] Registered new tserver with Master: b547566264d54b3fb0bcd8465baac89c (127.5.83.3:46773)
I20260808 04:08:42.116246 30660 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.3@KRBTEST.COM'} at 127.5.83.3:43083
I20260808 04:08:42.116928 5452 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:08:42.133387 31329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:42.121359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34896 (local address 127.5.83.62:46615)
0808 04:08:42.121524 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:42.121528 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:42.122022 (+ 494us) server_negotiation.cc:407] Connection header received
0808 04:08:42.122176 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:42.122181 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:42.122242 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:42.122319 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:08:42.123683 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:42.124409 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:42.125729 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:42.126129 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:42.126236 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:42.128623 (+ 2387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:08:42.128628 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:08:42.128631 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:08:42.128666 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:08:42.130852 (+ 2186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:08:42.131483 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:08:42.131489 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:08:42.131492 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:08:42.131618 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:08:42.131983 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:08:42.131989 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:08:42.131992 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:08:42.132290 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:08:42.132514 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:42.132852 (+ 338us) server_negotiation.cc:299] Negotiation successful
0808 04:08:42.133074 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 04:08:42.137233 30659 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34896:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:08:42.143975 30659 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:08:42.167176 31347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:42.155754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60784 (local address 127.5.83.1:40165)
0808 04:08:42.156422 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:42.156430 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:42.157581 (+ 1151us) server_negotiation.cc:407] Connection header received
0808 04:08:42.157682 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:42.157689 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:42.157902 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:42.158091 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:08:42.158935 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:42.160584 (+ 1649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:42.164101 (+ 3517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:42.165584 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:42.165697 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:42.165920 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:42.166650 (+ 730us) server_negotiation.cc:299] Negotiation successful
0808 04:08:42.166823 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":332,"threads_started":1}
I20260808 04:08:42.168341 31343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:42.155342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:40165 (local address 127.0.0.1:60784)
0808 04:08:42.157313 (+ 1971us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:42.157329 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:08:42.157462 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:08:42.158355 (+ 893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:08:42.158361 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:08:42.158384 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:08:42.158761 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:42.158772 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:42.160752 (+ 1980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:42.160757 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:42.163899 (+ 3142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:42.163913 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:42.166118 (+ 2205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:42.166125 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:42.166380 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:42.166441 (+ 61us) client_negotiation.cc:769] Sending connection context
0808 04:08:42.167697 (+ 1256us) client_negotiation.cc:240] Negotiation successful
0808 04:08:42.167906 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1853,"mutex_wait_us":665,"thread_start_us":977,"threads_started":1}
I20260808 04:08:42.168731 31344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:42.155180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37148 (local address 127.5.83.2:35969)
0808 04:08:42.155795 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:42.155803 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:42.156772 (+ 969us) server_negotiation.cc:407] Connection header received
0808 04:08:42.157033 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:42.157041 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:42.157291 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:42.157474 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:08:42.158518 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:42.161284 (+ 2766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:42.164902 (+ 3618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:42.166334 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:42.166430 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:42.166622 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:42.167228 (+ 606us) server_negotiation.cc:299] Negotiation successful
0808 04:08:42.167393 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":304,"threads_started":1}
I20260808 04:08:42.169555 31346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:42.155043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.2:35969 (local address 127.0.0.1:37148)
0808 04:08:42.156623 (+ 1580us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:42.156658 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0808 04:08:42.156816 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:08:42.157747 (+ 931us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:08:42.157754 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:08:42.157782 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:08:42.158291 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:42.158311 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:42.161472 (+ 3161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:42.161476 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:42.164701 (+ 3225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:42.164715 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:42.166822 (+ 2107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:42.166828 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:42.167015 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:42.167046 (+ 31us) client_negotiation.cc:769] Sending connection context
0808 04:08:42.169044 (+ 1998us) client_negotiation.cc:240] Negotiation successful
0808 04:08:42.169179 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1390,"mutex_wait_us":3,"thread_start_us":1995,"threads_started":1}
I20260808 04:08:42.172184 30849 tablet_service.cc:1467] Processing CreateTablet for tablet d4421234891c4995ba4df37dc92eedb8 (DEFAULT_TABLE table=test-table [id=0ba5eead257d4563b20d35ca5399b95f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:08:42.172869 31035 tablet_service.cc:1467] Processing CreateTablet for tablet d4421234891c4995ba4df37dc92eedb8 (DEFAULT_TABLE table=test-table [id=0ba5eead257d4563b20d35ca5399b95f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:08:42.173705 30849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4421234891c4995ba4df37dc92eedb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:08:42.173805 31348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:42.158410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.3:46773 (local address 127.0.0.1:58478)
0808 04:08:42.162606 (+ 4196us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:42.162620 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:08:42.162744 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:08:42.163576 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:08:42.163580 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:08:42.163605 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:08:42.164106 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:42.164118 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:42.166072 (+ 1954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:42.166077 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:42.168799 (+ 2722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:42.168821 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:42.173117 (+ 4296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:42.173123 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:42.173283 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:42.173315 (+ 32us) client_negotiation.cc:769] Sending connection context
0808 04:08:42.173398 (+ 83us) client_negotiation.cc:240] Negotiation successful
0808 04:08:42.173514 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4082,"thread_start_us":3904,"threads_started":1}
I20260808 04:08:42.174213 31035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4421234891c4995ba4df37dc92eedb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:08:42.174563 31349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:42.159024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58478 (local address 127.5.83.3:46773)
0808 04:08:42.162389 (+ 3365us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:42.162397 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:42.162713 (+ 316us) server_negotiation.cc:407] Connection header received
0808 04:08:42.162908 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:42.162915 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:42.163131 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:42.163309 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:08:42.164327 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:42.165859 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:42.168993 (+ 3134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:42.172609 (+ 3616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:42.172735 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:42.172960 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:42.174070 (+ 1110us) server_negotiation.cc:299] Negotiation successful
0808 04:08:42.174230 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3188,"thread_start_us":3016,"threads_started":1}
I20260808 04:08:42.177973 31236 tablet_service.cc:1467] Processing CreateTablet for tablet d4421234891c4995ba4df37dc92eedb8 (DEFAULT_TABLE table=test-table [id=0ba5eead257d4563b20d35ca5399b95f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:08:42.179260 31236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4421234891c4995ba4df37dc92eedb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:08:42.184461 31350 tablet_bootstrap.cc:492] T d4421234891c4995ba4df37dc92eedb8 P ec04c83d4a824de9989627e8e51747f8: Bootstrap starting.
I20260808 04:08:42.187798 31351 tablet_bootstrap.cc:492] T d4421234891c4995ba4df37dc92eedb8 P f5b3416abb7244e78131b28f7692aa33: Bootstrap starting.
I20260808 04:08:42.189218 31350 tablet_bootstrap.cc:654] T d4421234891c4995ba4df37dc92eedb8 P ec04c83d4a824de9989627e8e51747f8: Neither blocks nor log segments found. Creating new log.
I20260808 04:08:42.190631 31350 log.cc:824] T d4421234891c4995ba4df37dc92eedb8 P ec04c83d4a824de9989627e8e51747f8: Log is configured to *not* fsync() on all Append() calls
I20260808 04:08:42.191295 31351 tablet_bootstrap.cc:654] T d4421234891c4995ba4df37dc92eedb8 P f5b3416abb7244e78131b28f7692aa33: Neither blocks nor log segments found. Creating new log.
I20260808 04:08:42.192378 31351 log.cc:824] T d4421234891c4995ba4df37dc92eedb8 P f5b3416abb7244e78131b28f7692aa33: Log is configured to *not* fsync() on all Append() calls
I20260808 04:08:42.192895 31356 tablet_bootstrap.cc:492] T d4421234891c4995ba4df37dc92eedb8 P b547566264d54b3fb0bcd8465baac89c: Bootstrap starting.
I20260808 04:08:42.193895 31350 tablet_bootstrap.cc:492] T d4421234891c4995ba4df37dc92eedb8 P ec04c83d4a824de9989627e8e51747f8: No bootstrap required, opened a new log
I20260808 04:08:42.194130 31350 ts_tablet_manager.cc:1397] T d4421234891c4995ba4df37dc92eedb8 P ec04c83d4a824de9989627e8e51747f8: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.005s
I20260808 04:08:42.196374 31356 tablet_bootstrap.cc:654] T d4421234891c4995ba4df37dc92eedb8 P b547566264d54b3fb0bcd8465baac89c: Neither blocks nor log segments found. Creating new log.
I20260808 04:08:42.197579 31356 log.cc:824] T d4421234891c4995ba4df37dc92eedb8 P b547566264d54b3fb0bcd8465baac89c: Log is configured to *not* fsync() on all Append() calls
I20260808 04:08:42.200140 31356 tablet_bootstrap.cc:492] T d4421234891c4995ba4df37dc92eedb8 P b547566264d54b3fb0bcd8465baac89c: No bootstrap required, opened a new log
I20260808 04:08:42.200359 31356 ts_tablet_manager.cc:1397] T d4421234891c4995ba4df37dc92eedb8 P b547566264d54b3fb0bcd8465baac89c: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260808 04:08:42.200572 31351 tablet_bootstrap.cc:492] T d4421234891c4995ba4df37dc92eedb8 P f5b3416abb7244e78131b28f7692aa33: No bootstrap required, opened a new log
I20260808 04:08:42.200840 31351 ts_tablet_manager.cc:1397] T d4421234891c4995ba4df37dc92eedb8 P f5b3416abb7244e78131b28f7692aa33: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.003s
I20260808 04:08:42.202706 31350 raft_consensus.cc:348] T d4421234891c4995ba4df37dc92eedb8 P ec04c83d4a824de9989627e8e51747f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5b3416abb7244e78131b28f7692aa33" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 35969 } } peers { permanent_uuid: "ec04c83d4a824de9989627e8e51747f8" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 40165 } } peers { permanent_uuid: "b547566264d54b3fb0bcd8465baac89c" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 46773 } }
I20260808 04:08:42.203141 31350 raft_consensus.cc:374] T d4421234891c4995ba4df37dc92eedb8 P ec04c83d4a824de9989627e8e51747f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:08:42.203238 31350 raft_consensus.cc:729] T d4421234891c4995ba4df37dc92eedb8 P ec04c83d4a824de9989627e8e51747f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec04c83d4a824de9989627e8e51747f8, State: Initialized, Role: FOLLOWER
I20260808 04:08:42.203869 31350 consensus_queue.cc:260] T d4421234891c4995ba4df37dc92eedb8 P ec04c83d4a824de9989627e8e51747f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5b3416abb7244e78131b28f7692aa33" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 35969 } } peers { permanent_uuid: "ec04c83d4a824de9989627e8e51747f8" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 40165 } } peers { permanent_uuid: "b547566264d54b3fb0bcd8465baac89c" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 46773 } }
I20260808 04:08:42.205591 31350 ts_tablet_manager.cc:1428] T d4421234891c4995ba4df37dc92eedb8 P ec04c83d4a824de9989627e8e51747f8: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260808 04:08:42.208065 31356 raft_consensus.cc:348] T d4421234891c4995ba4df37dc92eedb8 P b547566264d54b3fb0bcd8465baac89c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5b3416abb7244e78131b28f7692aa33" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 35969 } } peers { permanent_uuid: "ec04c83d4a824de9989627e8e51747f8" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 40165 } } peers { permanent_uuid: "b547566264d54b3fb0bcd8465baac89c" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 46773 } }
I20260808 04:08:42.208437 31356 raft_consensus.cc:374] T d4421234891c4995ba4df37dc92eedb8 P b547566264d54b3fb0bcd8465baac89c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:08:42.208526 31356 raft_consensus.cc:729] T d4421234891c4995ba4df37dc92eedb8 P b547566264d54b3fb0bcd8465baac89c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b547566264d54b3fb0bcd8465baac89c, State: Initialized, Role: FOLLOWER
I20260808 04:08:42.209101 31356 consensus_queue.cc:260] T d4421234891c4995ba4df37dc92eedb8 P b547566264d54b3fb0bcd8465baac89c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5b3416abb7244e78131b28f7692aa33" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 35969 } } peers { permanent_uuid: "ec04c83d4a824de9989627e8e51747f8" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 40165 } } peers { permanent_uuid: "b547566264d54b3fb0bcd8465baac89c" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 46773 } }
I20260808 04:08:42.210052 31326 heartbeater.cc:503] Master 127.5.83.62:46615 was elected leader, sending a full tablet report...
I20260808 04:08:42.210636 31356 ts_tablet_manager.cc:1428] T d4421234891c4995ba4df37dc92eedb8 P b547566264d54b3fb0bcd8465baac89c: Time spent starting tablet: real 0.010s user 0.006s sys 0.004s
I20260808 04:08:42.211548 31351 raft_consensus.cc:348] T d4421234891c4995ba4df37dc92eedb8 P f5b3416abb7244e78131b28f7692aa33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5b3416abb7244e78131b28f7692aa33" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 35969 } } peers { permanent_uuid: "ec04c83d4a824de9989627e8e51747f8" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 40165 } } peers { permanent_uuid: "b547566264d54b3fb0bcd8465baac89c" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 46773 } }
I20260808 04:08:42.211988 31351 raft_consensus.cc:374] T d4421234891c4995ba4df37dc92eedb8 P f5b3416abb7244e78131b28f7692aa33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:08:42.212098 31351 raft_consensus.cc:729] T d4421234891c4995ba4df37dc92eedb8 P f5b3416abb7244e78131b28f7692aa33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5b3416abb7244e78131b28f7692aa33, State: Initialized, Role: FOLLOWER
I20260808 04:08:42.212659 31351 consensus_queue.cc:260] T d4421234891c4995ba4df37dc92eedb8 P f5b3416abb7244e78131b28f7692aa33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5b3416abb7244e78131b28f7692aa33" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 35969 } } peers { permanent_uuid: "ec04c83d4a824de9989627e8e51747f8" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 40165 } } peers { permanent_uuid: "b547566264d54b3fb0bcd8465baac89c" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 46773 } }
I20260808 04:08:42.213624 31124 heartbeater.cc:503] Master 127.5.83.62:46615 was elected leader, sending a full tablet report...
I20260808 04:08:42.214277 31351 ts_tablet_manager.cc:1428] T d4421234891c4995ba4df37dc92eedb8 P f5b3416abb7244e78131b28f7692aa33: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260808 04:08:42.228051 31359 raft_consensus.cc:482] T d4421234891c4995ba4df37dc92eedb8 P ec04c83d4a824de9989627e8e51747f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:08:42.228302 31359 raft_consensus.cc:504] T d4421234891c4995ba4df37dc92eedb8 P ec04c83d4a824de9989627e8e51747f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5b3416abb7244e78131b28f7692aa33" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 35969 } } peers { permanent_uuid: "ec04c83d4a824de9989627e8e51747f8" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 40165 } } peers { permanent_uuid: "b547566264d54b3fb0bcd8465baac89c" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 46773 } }
I20260808 04:08:42.229727 31359 leader_election.cc:288] T d4421234891c4995ba4df37dc92eedb8 P ec04c83d4a824de9989627e8e51747f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f5b3416abb7244e78131b28f7692aa33 (127.5.83.2:35969), b547566264d54b3fb0bcd8465baac89c (127.5.83.3:46773)
I20260808 04:08:42.236794 31344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:42.230059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.1:36549 (local address 127.5.83.2:35969)
0808 04:08:42.230247 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:42.230254 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:42.230826 (+ 572us) server_negotiation.cc:407] Connection header received
0808 04:08:42.231001 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:42.231005 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:42.231059 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:42.231130 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:08:42.231662 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:42.232411 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:42.234619 (+ 2208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:42.235868 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:42.235989 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:42.236062 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:42.236498 (+ 436us) server_negotiation.cc:299] Negotiation successful
0808 04:08:42.236598 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 04:08:42.237004 31364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:42.230033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.2:35969 (local address 127.5.83.1:36549)
0808 04:08:42.230724 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:42.230739 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:08:42.230878 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:08:42.231161 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:08:42.231166 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:08:42.231197 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:08:42.231533 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:42.231542 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:42.232536 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:42.232541 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:42.234460 (+ 1919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:42.234476 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:42.236107 (+ 1631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:42.236113 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:42.236303 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:42.236338 (+ 35us) client_negotiation.cc:769] Sending connection context
0808 04:08:42.236447 (+ 109us) client_negotiation.cc:240] Negotiation successful
0808 04:08:42.236574 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"mutex_wait_us":2,"thread_start_us":421,"threads_started":1}
I20260808 04:08:42.237828 31062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4421234891c4995ba4df37dc92eedb8" candidate_uuid: "ec04c83d4a824de9989627e8e51747f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5b3416abb7244e78131b28f7692aa33" is_pre_election: true
I20260808 04:08:42.238246 31062 raft_consensus.cc:2454] T d4421234891c4995ba4df37dc92eedb8 P f5b3416abb7244e78131b28f7692aa33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec04c83d4a824de9989627e8e51747f8 in term 0.
I20260808 04:08:42.238814 30772 leader_election.cc:302] T d4421234891c4995ba4df37dc92eedb8 P ec04c83d4a824de9989627e8e51747f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ec04c83d4a824de9989627e8e51747f8, f5b3416abb7244e78131b28f7692aa33; no voters:
I20260808 04:08:42.239174 31359 raft_consensus.cc:2781] T d4421234891c4995ba4df37dc92eedb8 P ec04c83d4a824de9989627e8e51747f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:08:42.239328 31359 raft_consensus.cc:482] T d4421234891c4995ba4df37dc92eedb8 P ec04c83d4a824de9989627e8e51747f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:08:42.239444 31349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:42.230186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.1:47851 (local address 127.5.83.3:46773)
0808 04:08:42.230350 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:42.230355 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:42.233799 (+ 3444us) server_negotiation.cc:407] Connection header received
0808 04:08:42.233940 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:42.233946 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:42.234001 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:42.234074 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:08:42.234651 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:42.235798 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:42.237775 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:42.238587 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:42.238697 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:42.238792 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:42.239105 (+ 313us) server_negotiation.cc:299] Negotiation successful
0808 04:08:42.239218 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 04:08:42.239447 31359 raft_consensus.cc:3037] T d4421234891c4995ba4df37dc92eedb8 P ec04c83d4a824de9989627e8e51747f8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:08:42.239629 31365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:42.230100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.3:46773 (local address 127.5.83.1:47851)
0808 04:08:42.233687 (+ 3587us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:42.233704 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:08:42.233822 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:08:42.234115 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:08:42.234120 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:08:42.234144 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:08:42.234499 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:42.234507 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:42.235928 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:42.235933 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:42.237633 (+ 1700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:42.237645 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:42.238814 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:42.238821 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:42.238947 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:42.238969 (+ 22us) client_negotiation.cc:769] Sending connection context
0808 04:08:42.239058 (+ 89us) client_negotiation.cc:240] Negotiation successful
0808 04:08:42.239183 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3469,"thread_start_us":3296,"threads_started":1}
I20260808 04:08:42.240553 31261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4421234891c4995ba4df37dc92eedb8" candidate_uuid: "ec04c83d4a824de9989627e8e51747f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b547566264d54b3fb0bcd8465baac89c" is_pre_election: true
I20260808 04:08:42.240969 31261 raft_consensus.cc:2454] T d4421234891c4995ba4df37dc92eedb8 P b547566264d54b3fb0bcd8465baac89c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec04c83d4a824de9989627e8e51747f8 in term 0.
I20260808 04:08:42.241986 31359 raft_consensus.cc:504] T d4421234891c4995ba4df37dc92eedb8 P ec04c83d4a824de9989627e8e51747f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5b3416abb7244e78131b28f7692aa33" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 35969 } } peers { permanent_uuid: "ec04c83d4a824de9989627e8e51747f8" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 40165 } } peers { permanent_uuid: "b547566264d54b3fb0bcd8465baac89c" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 46773 } }
I20260808 04:08:42.242519 31359 leader_election.cc:288] T d4421234891c4995ba4df37dc92eedb8 P ec04c83d4a824de9989627e8e51747f8 [CANDIDATE]: Term 1 election: Requested vote from peers f5b3416abb7244e78131b28f7692aa33 (127.5.83.2:35969), b547566264d54b3fb0bcd8465baac89c (127.5.83.3:46773)
I20260808 04:08:42.242955 31062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4421234891c4995ba4df37dc92eedb8" candidate_uuid: "ec04c83d4a824de9989627e8e51747f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5b3416abb7244e78131b28f7692aa33"
I20260808 04:08:42.242990 31261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4421234891c4995ba4df37dc92eedb8" candidate_uuid: "ec04c83d4a824de9989627e8e51747f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b547566264d54b3fb0bcd8465baac89c"
I20260808 04:08:42.243149 31261 raft_consensus.cc:3037] T d4421234891c4995ba4df37dc92eedb8 P b547566264d54b3fb0bcd8465baac89c [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:08:42.243149 31062 raft_consensus.cc:3037] T d4421234891c4995ba4df37dc92eedb8 P f5b3416abb7244e78131b28f7692aa33 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:08:42.246340 31062 raft_consensus.cc:2454] T d4421234891c4995ba4df37dc92eedb8 P f5b3416abb7244e78131b28f7692aa33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec04c83d4a824de9989627e8e51747f8 in term 1.
I20260808 04:08:42.246353 31261 raft_consensus.cc:2454] T d4421234891c4995ba4df37dc92eedb8 P b547566264d54b3fb0bcd8465baac89c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec04c83d4a824de9989627e8e51747f8 in term 1.
I20260808 04:08:42.246702 30772 leader_election.cc:302] T d4421234891c4995ba4df37dc92eedb8 P ec04c83d4a824de9989627e8e51747f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ec04c83d4a824de9989627e8e51747f8, f5b3416abb7244e78131b28f7692aa33; no voters:
I20260808 04:08:42.247000 31359 raft_consensus.cc:2781] T d4421234891c4995ba4df37dc92eedb8 P ec04c83d4a824de9989627e8e51747f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:08:42.247391 31359 raft_consensus.cc:686] T d4421234891c4995ba4df37dc92eedb8 P ec04c83d4a824de9989627e8e51747f8 [term 1 LEADER]: Becoming Leader. State: Replica: ec04c83d4a824de9989627e8e51747f8, State: Running, Role: LEADER
I20260808 04:08:42.247843 31359 consensus_queue.cc:237] T d4421234891c4995ba4df37dc92eedb8 P ec04c83d4a824de9989627e8e51747f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5b3416abb7244e78131b28f7692aa33" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 35969 } } peers { permanent_uuid: "ec04c83d4a824de9989627e8e51747f8" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 40165 } } peers { permanent_uuid: "b547566264d54b3fb0bcd8465baac89c" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 46773 } }
I20260808 04:08:42.252137 30659 catalog_manager.cc:5495] T d4421234891c4995ba4df37dc92eedb8 P ec04c83d4a824de9989627e8e51747f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec04c83d4a824de9989627e8e51747f8 (127.5.83.1). New cstate: current_term: 1 leader_uuid: "ec04c83d4a824de9989627e8e51747f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5b3416abb7244e78131b28f7692aa33" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 35969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec04c83d4a824de9989627e8e51747f8" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 40165 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b547566264d54b3fb0bcd8465baac89c" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 46773 } health_report { overall_health: UNKNOWN } } }
I20260808 04:08:42.289033 31347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:42.283692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60796 (local address 127.5.83.1:40165)
0808 04:08:42.283893 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:42.283909 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:42.283974 (+ 65us) server_negotiation.cc:407] Connection header received
0808 04:08:42.284190 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:42.284193 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:42.284284 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:42.284410 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:08:42.284789 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:42.285504 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:42.287048 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:42.287457 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:42.287585 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:42.288116 (+ 531us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:08:42.288591 (+ 475us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:08:42.288696 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:42.288773 (+ 77us) server_negotiation.cc:299] Negotiation successful
0808 04:08:42.288836 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 04:08:42.296666 31261 raft_consensus.cc:1261] T d4421234891c4995ba4df37dc92eedb8 P b547566264d54b3fb0bcd8465baac89c [term 1 FOLLOWER]: Refusing update from remote peer ec04c83d4a824de9989627e8e51747f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:08:42.296666 31062 raft_consensus.cc:1261] T d4421234891c4995ba4df37dc92eedb8 P f5b3416abb7244e78131b28f7692aa33 [term 1 FOLLOWER]: Refusing update from remote peer ec04c83d4a824de9989627e8e51747f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:08:42.297760 31370 consensus_queue.cc:1035] T d4421234891c4995ba4df37dc92eedb8 P ec04c83d4a824de9989627e8e51747f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b547566264d54b3fb0bcd8465baac89c" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 46773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:08:42.298134 31359 consensus_queue.cc:1035] T d4421234891c4995ba4df37dc92eedb8 P ec04c83d4a824de9989627e8e51747f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5b3416abb7244e78131b28f7692aa33" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 35969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:08:42.314441 31379 mvcc.cc:205] Tried to move back new op lower bound from 7316120052916944896 to 7316120052730372096. Current Snapshot: MvccSnapshot[applied={T|T < 7316120052916944896}]
I20260808 04:08:42.315929 31378 mvcc.cc:205] Tried to move back new op lower bound from 7316120052916944896 to 7316120052730372096. Current Snapshot: MvccSnapshot[applied={T|T < 7316120052916944896}]
I20260808 04:08:42.327203 30659 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34896:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:08:42.327471 30659 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34896:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:08:42.330871 30659 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6f9c51b72c6e4f0a8d53b56e8559674a: Sending DeleteTablet for 3 replicas of tablet d4421234891c4995ba4df37dc92eedb8
I20260808 04:08:42.332070 30849 tablet_service.cc:1514] Processing DeleteTablet for tablet d4421234891c4995ba4df37dc92eedb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:08:42 UTC) from {username='kudu', principal='kudu/127.5.83.62@KRBTEST.COM'} at 127.0.0.1:60784
I20260808 04:08:42.332123 31236 tablet_service.cc:1514] Processing DeleteTablet for tablet d4421234891c4995ba4df37dc92eedb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:08:42 UTC) from {username='kudu', principal='kudu/127.5.83.62@KRBTEST.COM'} at 127.0.0.1:58478
I20260808 04:08:42.332108 31035 tablet_service.cc:1514] Processing DeleteTablet for tablet d4421234891c4995ba4df37dc92eedb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:08:42 UTC) from {username='kudu', principal='kudu/127.5.83.62@KRBTEST.COM'} at 127.0.0.1:37148
I20260808 04:08:42.333405 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 30750
I20260808 04:08:42.351338 31394 tablet_replica.cc:330] T d4421234891c4995ba4df37dc92eedb8 P ec04c83d4a824de9989627e8e51747f8: stopping tablet replica
I20260808 04:08:42.351836 31394 raft_consensus.cc:2229] T d4421234891c4995ba4df37dc92eedb8 P ec04c83d4a824de9989627e8e51747f8 [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:08:42.352523 31394 raft_consensus.cc:2258] T d4421234891c4995ba4df37dc92eedb8 P ec04c83d4a824de9989627e8e51747f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:08:42.353652 31395 tablet_replica.cc:330] T d4421234891c4995ba4df37dc92eedb8 P b547566264d54b3fb0bcd8465baac89c: stopping tablet replica
I20260808 04:08:42.353961 31395 raft_consensus.cc:2229] T d4421234891c4995ba4df37dc92eedb8 P b547566264d54b3fb0bcd8465baac89c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:08:42.354283 31396 tablet_replica.cc:330] T d4421234891c4995ba4df37dc92eedb8 P f5b3416abb7244e78131b28f7692aa33: stopping tablet replica
I20260808 04:08:42.354460 31395 raft_consensus.cc:2258] T d4421234891c4995ba4df37dc92eedb8 P b547566264d54b3fb0bcd8465baac89c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:08:42.354667 31396 raft_consensus.cc:2229] T d4421234891c4995ba4df37dc92eedb8 P f5b3416abb7244e78131b28f7692aa33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:08:42.356639 31395 ts_tablet_manager.cc:1905] T d4421234891c4995ba4df37dc92eedb8 P b547566264d54b3fb0bcd8465baac89c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:08:42.356766 31394 ts_tablet_manager.cc:1905] T d4421234891c4995ba4df37dc92eedb8 P ec04c83d4a824de9989627e8e51747f8: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 08 04:08:42 dist-test-slave-1l3l krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.1@KRBTEST.COM
I20260808 04:08:42.361439 31396 raft_consensus.cc:2258] T d4421234891c4995ba4df37dc92eedb8 P f5b3416abb7244e78131b28f7692aa33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:08:42.363343 31395 ts_tablet_manager.cc:1918] T d4421234891c4995ba4df37dc92eedb8 P b547566264d54b3fb0bcd8465baac89c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:08:42.363510 31395 log.cc:1196] T d4421234891c4995ba4df37dc92eedb8 P b547566264d54b3fb0bcd8465baac89c: Deleting WAL directory at /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/minicluster-data/ts-2/wal/wals/d4421234891c4995ba4df37dc92eedb8
I20260808 04:08:42.363579 31396 ts_tablet_manager.cc:1905] T d4421234891c4995ba4df37dc92eedb8 P f5b3416abb7244e78131b28f7692aa33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:08:42.364104 31395 ts_tablet_manager.cc:1939] T d4421234891c4995ba4df37dc92eedb8 P b547566264d54b3fb0bcd8465baac89c: Deleting consensus metadata
I20260808 04:08:42.365911 30637 catalog_manager.cc:4841] TS b547566264d54b3fb0bcd8465baac89c (127.5.83.3:46773): tablet d4421234891c4995ba4df37dc92eedb8 (table test-table [id=0ba5eead257d4563b20d35ca5399b95f]) successfully deleted
I20260808 04:08:42.367580 31347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:42.334490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60812 (local address 127.5.83.1:40165)
0808 04:08:42.334657 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:42.334664 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:42.345039 (+ 10375us) server_negotiation.cc:407] Connection header received
0808 04:08:42.345309 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:42.345315 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:42.345414 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:42.345488 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:08:42.347437 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:42.348815 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:42.353223 (+ 4408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:42.353902 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:42.354033 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:42.359885 (+ 5852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:08:42.359908 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:08:42.359923 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:08:42.359975 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:08:42.364219 (+ 4244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:08:42.365087 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:08:42.365095 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:08:42.365098 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:08:42.365292 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:08:42.365746 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:08:42.365751 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:08:42.365754 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:08:42.366233 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:08:42.366490 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:42.366978 (+ 488us) server_negotiation.cc:299] Negotiation successful
0808 04:08:42.367218 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 04:08:42.368268 30930 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:08:42.370276 31396 ts_tablet_manager.cc:1918] T d4421234891c4995ba4df37dc92eedb8 P f5b3416abb7244e78131b28f7692aa33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:08:42.370420 31396 log.cc:1196] T d4421234891c4995ba4df37dc92eedb8 P f5b3416abb7244e78131b28f7692aa33: Deleting WAL directory at /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161826605655-5452-0/minicluster-data/ts-1/wal/wals/d4421234891c4995ba4df37dc92eedb8
I20260808 04:08:42.370932 31396 ts_tablet_manager.cc:1939] T d4421234891c4995ba4df37dc92eedb8 P f5b3416abb7244e78131b28f7692aa33: Deleting consensus metadata
I20260808 04:08:42.372699 30633 catalog_manager.cc:4841] TS f5b3416abb7244e78131b28f7692aa33 (127.5.83.2:35969): tablet d4421234891c4995ba4df37dc92eedb8 (table test-table [id=0ba5eead257d4563b20d35ca5399b95f]) successfully deleted
I20260808 04:08:42.464779 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 30750
W20260808 04:08:42.484562 30637 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.83.1:40165 (error 108)
I20260808 04:08:42.485392 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 30944
I20260808 04:08:42.486707 31348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:42.486215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:40165 (local address 127.0.0.1:60814)
0808 04:08:42.486444 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:42.486555 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.1:40165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
W20260808 04:08:42.486967 30637 catalog_manager.cc:4584] TS ec04c83d4a824de9989627e8e51747f8 (127.5.83.1:40165): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d4421234891c4995ba4df37dc92eedb8: Network error: Client connection negotiation failed: client connection to 127.5.83.1:40165: connect: Connection refused (error 111)
Aug 08 04:08:42 dist-test-slave-1l3l krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.2@KRBTEST.COM
I20260808 04:08:42.500289 31344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:42.486188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37156 (local address 127.5.83.2:35969)
0808 04:08:42.486353 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:42.486358 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:42.486376 (+ 18us) server_negotiation.cc:407] Connection header received
0808 04:08:42.486587 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:42.486591 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:42.486652 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:42.486744 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:08:42.488257 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:42.489388 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:42.490579 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:42.491193 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:42.491316 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:42.495339 (+ 4023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:08:42.495353 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:08:42.495367 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:08:42.495425 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:08:42.497941 (+ 2516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:08:42.498552 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:08:42.498558 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:08:42.498561 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:08:42.498672 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:08:42.498934 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:08:42.498937 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:08:42.498939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:08:42.499284 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:08:42.499503 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:42.499670 (+ 167us) server_negotiation.cc:299] Negotiation successful
0808 04:08:42.499981 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 04:08:42.500911 31118 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:08:42.540539 31348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:42.540073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:40165 (local address 127.0.0.1:60822)
0808 04:08:42.540263 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:42.540362 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.1:40165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260808 04:08:42.541481 31348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:42.541164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:40165 (local address 127.0.0.1:60838)
0808 04:08:42.541277 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:42.541341 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.1:40165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260808 04:08:42.588446 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 30944
I20260808 04:08:42.602015 31348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:42.601516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:40165 (local address 127.0.0.1:60852)
0808 04:08:42.601744 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:42.601845 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.1:40165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260808 04:08:42.602985 31348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:42.602659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:40165 (local address 127.0.0.1:60858)
0808 04:08:42.602808 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:42.602876 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.1:40165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260808 04:08:42.607123 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 31131
Aug 08 04:08:42 dist-test-slave-1l3l krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.3@KRBTEST.COM
I20260808 04:08:42.621901 31349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:42.608081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58486 (local address 127.5.83.3:46773)
0808 04:08:42.608258 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:42.608264 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:42.608364 (+ 100us) server_negotiation.cc:407] Connection header received
0808 04:08:42.608565 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:42.608569 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:42.608645 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:42.608725 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:08:42.609990 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:42.610711 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:42.612004 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:42.612427 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:42.612538 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:42.616831 (+ 4293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:08:42.616846 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:08:42.616863 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:08:42.616905 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:08:42.619538 (+ 2633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:08:42.620166 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:08:42.620173 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:08:42.620176 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:08:42.620309 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:08:42.620603 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:08:42.620605 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:08:42.620607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:08:42.620992 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:08:42.621207 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:42.621477 (+ 270us) server_negotiation.cc:299] Negotiation successful
0808 04:08:42.621660 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 04:08:42.622515 31318 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:08:42.683447 31348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:42.682975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:40165 (local address 127.0.0.1:60862)
0808 04:08:42.683173 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:42.683286 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.1:40165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260808 04:08:42.684551 31348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:42.684177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:40165 (local address 127.0.0.1:60874)
0808 04:08:42.684356 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:42.684439 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.1:40165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260808 04:08:42.708791 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 31131
I20260808 04:08:42.728664 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 30611
I20260808 04:08:42.729756 30707 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:08:42.835783 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 30611
2026-08-08T04:08:42Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4238 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-08T04:08:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:08:42Z Disabled control of system clock
I20260808 04:08:42.922873 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161826605655-5452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161826605655-5452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161826605655-5452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:42833
--webserver_interface=127.5.83.62
--webserver_port=0
--builtin_ntp_servers=127.5.83.20:36023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.62:42833
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 04:08:43.241647 31410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:08:43.241935 31410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:08:43.242070 31410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:08:43.249265 31410 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:08:43.249387 31410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:08:43.249449 31410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:08:43.249531 31410 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:08:43.249593 31410 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:08:43.258661 31410 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:36023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161826605655-5452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161826605655-5452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.62:42833
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:42833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161826605655-5452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:08:43.259958 31410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:08:43.261749 31410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:08:43.272799 31419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:08:43.272858 31418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:08:43.272862 31422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:08:43.273732 31410 server_base.cc:1034] running on GCE node
I20260808 04:08:43.274392 31410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:08:43.275691 31410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:08:43.276917 31410 hybrid_clock.cc:648] HybridClock initialized: now 1786162123276870 us; error 41 us; skew 500 ppm
I20260808 04:08:43.277233 31410 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 04:08:43.284006 31410 webserver.cc:460] Webserver started at http://127.5.83.62:42937/ using document root <none> and password file <none>
I20260808 04:08:43.284638 31410 fs_manager.cc:359] Metadata directory not provided
I20260808 04:08:43.284740 31410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:08:43.285043 31410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:08:43.287823 31410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161826605655-5452-0/minicluster-data/master-0/data/instance:
uuid: "eb7da5edad5c436e884c6e3fe1cf80f6"
format_stamp: "Formatted at 2026-08-08 04:08:43 on dist-test-slave-1l3l"
server_key: "528b7d5888fc2da18b9f2c48b1d13cba"
server_key_iv: "0a15952c606b239c1506f507c155e4df"
server_key_version: "encryptionkey@0"
I20260808 04:08:43.288599 31410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161826605655-5452-0/minicluster-data/master-0/wal/instance:
uuid: "eb7da5edad5c436e884c6e3fe1cf80f6"
format_stamp: "Formatted at 2026-08-08 04:08:43 on dist-test-slave-1l3l"
server_key: "528b7d5888fc2da18b9f2c48b1d13cba"
server_key_iv: "0a15952c606b239c1506f507c155e4df"
server_key_version: "encryptionkey@0"
I20260808 04:08:43.293371 31410 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 04:08:43.296645 31429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:08:43.297825 31410 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 04:08:43.297977 31410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161826605655-5452-0/minicluster-data/master-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161826605655-5452-0/minicluster-data/master-0/wal
uuid: "eb7da5edad5c436e884c6e3fe1cf80f6"
format_stamp: "Formatted at 2026-08-08 04:08:43 on dist-test-slave-1l3l"
server_key: "528b7d5888fc2da18b9f2c48b1d13cba"
server_key_iv: "0a15952c606b239c1506f507c155e4df"
server_key_version: "encryptionkey@0"
I20260808 04:08:43.298159 31410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161826605655-5452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161826605655-5452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161826605655-5452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:08:43.334311 31410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:08:43.335083 31410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:08:43.335351 31410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:08:43.351649 31410 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.62:42833
I20260808 04:08:43.351675 31496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.62:42833 every 8 connection(s)
I20260808 04:08:43.353099 31410 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
I20260808 04:08:43.353400 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 31410
I20260808 04:08:43.353597 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161826605655-5452-0/minicluster-data/master-0/wal/instance
I20260808 04:08:43.354247 5452 external_mini_cluster.cc:1428] Setting key 78a15772a2d6078ba1b506629bfb1690
I20260808 04:08:43.357751 31497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:08:43.369292 31502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:43.356144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52192 (local address 127.5.83.62:42833)
0808 04:08:43.357187 (+ 1043us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:43.357201 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:43.357257 (+ 56us) server_negotiation.cc:407] Connection header received
0808 04:08:43.359506 (+ 2249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:43.359532 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:43.360111 (+ 579us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:43.360528 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:08:43.361059 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:43.363077 (+ 2018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:43.364437 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:43.365398 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:43.365580 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:43.366585 (+ 1005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:08:43.366594 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:08:43.366609 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:08:43.367300 (+ 691us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:08:43.367403 (+ 103us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:08:43.367439 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:08:43.367561 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:43.367728 (+ 167us) server_negotiation.cc:299] Negotiation successful
0808 04:08:43.368042 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":348,"threads_started":1}
I20260808 04:08:43.371325 31497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb7da5edad5c436e884c6e3fe1cf80f6: Bootstrap starting.
I20260808 04:08:43.374148 31497 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb7da5edad5c436e884c6e3fe1cf80f6: Neither blocks nor log segments found. Creating new log.
I20260808 04:08:43.375227 31497 log.cc:824] T 00000000000000000000000000000000 P eb7da5edad5c436e884c6e3fe1cf80f6: Log is configured to *not* fsync() on all Append() calls
I20260808 04:08:43.377676 31497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb7da5edad5c436e884c6e3fe1cf80f6: No bootstrap required, opened a new log
I20260808 04:08:43.384847 31497 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb7da5edad5c436e884c6e3fe1cf80f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb7da5edad5c436e884c6e3fe1cf80f6" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 42833 } }
I20260808 04:08:43.385172 31497 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb7da5edad5c436e884c6e3fe1cf80f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:08:43.385267 31497 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb7da5edad5c436e884c6e3fe1cf80f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb7da5edad5c436e884c6e3fe1cf80f6, State: Initialized, Role: FOLLOWER
I20260808 04:08:43.385766 31497 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb7da5edad5c436e884c6e3fe1cf80f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb7da5edad5c436e884c6e3fe1cf80f6" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 42833 } }
I20260808 04:08:43.385977 31497 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb7da5edad5c436e884c6e3fe1cf80f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:08:43.386106 31497 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb7da5edad5c436e884c6e3fe1cf80f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:08:43.386245 31497 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb7da5edad5c436e884c6e3fe1cf80f6 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:08:43.388614 31497 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb7da5edad5c436e884c6e3fe1cf80f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb7da5edad5c436e884c6e3fe1cf80f6" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 42833 } }
I20260808 04:08:43.389061 31497 leader_election.cc:302] T 00000000000000000000000000000000 P eb7da5edad5c436e884c6e3fe1cf80f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb7da5edad5c436e884c6e3fe1cf80f6; no voters:
I20260808 04:08:43.389570 31497 leader_election.cc:288] T 00000000000000000000000000000000 P eb7da5edad5c436e884c6e3fe1cf80f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:08:43.389699 31504 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb7da5edad5c436e884c6e3fe1cf80f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:08:43.390571 31504 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb7da5edad5c436e884c6e3fe1cf80f6 [term 1 LEADER]: Becoming Leader. State: Replica: eb7da5edad5c436e884c6e3fe1cf80f6, State: Running, Role: LEADER
I20260808 04:08:43.391023 31504 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb7da5edad5c436e884c6e3fe1cf80f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb7da5edad5c436e884c6e3fe1cf80f6" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 42833 } }
I20260808 04:08:43.391249 31497 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb7da5edad5c436e884c6e3fe1cf80f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:08:43.393568 31505 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb7da5edad5c436e884c6e3fe1cf80f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb7da5edad5c436e884c6e3fe1cf80f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb7da5edad5c436e884c6e3fe1cf80f6" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 42833 } } }
I20260808 04:08:43.393982 31505 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb7da5edad5c436e884c6e3fe1cf80f6 [sys.catalog]: This master's current role is: LEADER
I20260808 04:08:43.393546 31508 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb7da5edad5c436e884c6e3fe1cf80f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb7da5edad5c436e884c6e3fe1cf80f6. Latest consensus state: current_term: 1 leader_uuid: "eb7da5edad5c436e884c6e3fe1cf80f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb7da5edad5c436e884c6e3fe1cf80f6" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 42833 } } }
I20260808 04:08:43.394438 31508 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb7da5edad5c436e884c6e3fe1cf80f6 [sys.catalog]: This master's current role is: LEADER
I20260808 04:08:43.398280 31522 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:08:43.402294 31522 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:08:43.409757 31522 catalog_manager.cc:1347] Generated new cluster ID: 348e948931fa46b99cd4a42292e95c88
I20260808 04:08:43.409862 31522 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:08:43.448374 31522 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:08:43.449658 31522 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:08:43.455245 31522 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb7da5edad5c436e884c6e3fe1cf80f6: Generated new TSK 0
I20260808 04:08:43.456060 31522 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:08:43.470044 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161826605655-5452-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161826605655-5452-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161826605655-5452-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.1:0
--local_ip_for_outbound_sockets=127.5.83.1
--webserver_interface=127.5.83.1
--webserver_port=0
--tserver_master_addrs=127.5.83.62:42833
--builtin_ntp_servers=127.5.83.20:36023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 04:08:43.794335 31538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:08:43.794611 31538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:08:43.794736 31538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:08:43.801785 31538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:08:43.801867 31538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:08:43.802026 31538 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.1
I20260808 04:08:43.810688 31538 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:36023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161826605655-5452-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161826605655-5452-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.83.1
--webserver_port=0
--tserver_master_addrs=127.5.83.62:42833
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.1
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161826605655-5452-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:08:43.811864 31538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:08:43.813519 31538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:08:43.825805 31545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:08:43.825866 31549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:08:43.825834 31546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:08:43.826537 31538 server_base.cc:1034] running on GCE node
I20260808 04:08:43.827072 31538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:08:43.827989 31538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:08:43.829191 31538 hybrid_clock.cc:648] HybridClock initialized: now 1786162123829152 us; error 32 us; skew 500 ppm
I20260808 04:08:43.829478 31538 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 04:08:43.831612 31538 webserver.cc:460] Webserver started at http://127.5.83.1:33037/ using document root <none> and password file <none>
I20260808 04:08:43.832284 31538 fs_manager.cc:359] Metadata directory not provided
I20260808 04:08:43.832372 31538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:08:43.832652 31538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:08:43.835392 31538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161826605655-5452-0/minicluster-data/ts-0/data/instance:
uuid: "c79d91a0da0e4ed1b8418b97a8e5802a"
format_stamp: "Formatted at 2026-08-08 04:08:43 on dist-test-slave-1l3l"
server_key: "4947899bfed7310f06eb584e5e148b64"
server_key_iv: "0c85bcd2046807da64e514917d7a56ac"
server_key_version: "encryptionkey@0"
I20260808 04:08:43.836162 31538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161826605655-5452-0/minicluster-data/ts-0/wal/instance:
uuid: "c79d91a0da0e4ed1b8418b97a8e5802a"
format_stamp: "Formatted at 2026-08-08 04:08:43 on dist-test-slave-1l3l"
server_key: "4947899bfed7310f06eb584e5e148b64"
server_key_iv: "0c85bcd2046807da64e514917d7a56ac"
server_key_version: "encryptionkey@0"
I20260808 04:08:43.841066 31538 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 04:08:43.844295 31559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:08:43.845510 31538 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260808 04:08:43.845640 31538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161826605655-5452-0/minicluster-data/ts-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161826605655-5452-0/minicluster-data/ts-0/wal
uuid: "c79d91a0da0e4ed1b8418b97a8e5802a"
format_stamp: "Formatted at 2026-08-08 04:08:43 on dist-test-slave-1l3l"
server_key: "4947899bfed7310f06eb584e5e148b64"
server_key_iv: "0c85bcd2046807da64e514917d7a56ac"
server_key_version: "encryptionkey@0"
I20260808 04:08:43.845810 31538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161826605655-5452-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161826605655-5452-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161826605655-5452-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:08:43.863822 31538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:08:43.864552 31538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:08:43.864781 31538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:08:43.865734 31538 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:08:43.867123 31538 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:08:43.867224 31538 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:08:43.867323 31538 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:08:43.867378 31538 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:08:43.898161 31538 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.1:33765
I20260808 04:08:43.898226 31721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.1:33765 every 8 connection(s)
I20260808 04:08:43.899431 31538 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
I20260808 04:08:43.908707 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 31538
I20260808 04:08:43.908864 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161826605655-5452-0/minicluster-data/ts-0/wal/instance
I20260808 04:08:43.909291 5452 external_mini_cluster.cc:1428] Setting key 636da3b1d4fd1b252cc17264743ea14e
I20260808 04:08:43.911391 31725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:43.901793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.1:53613 (local address 127.5.83.62:42833)
0808 04:08:43.902201 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:43.902205 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:43.903339 (+ 1134us) server_negotiation.cc:407] Connection header received
0808 04:08:43.905092 (+ 1753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:43.905099 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:43.905173 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:43.905268 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:08:43.906271 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:43.906974 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:43.908520 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:43.908954 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:43.909076 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:43.910303 (+ 1227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:08:43.910308 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:08:43.910311 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:08:43.910328 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:08:43.910367 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:08:43.910378 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:08:43.910465 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:43.910668 (+ 203us) server_negotiation.cc:299] Negotiation successful
0808 04:08:43.910743 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":218,"threads_started":1}
I20260808 04:08:43.912146 31726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:43.902313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:42833 (local address 127.5.83.1:53613)
0808 04:08:43.903195 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:43.903230 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0808 04:08:43.904828 (+ 1598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:08:43.905495 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:08:43.905503 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:08:43.905579 (+ 76us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:08:43.906111 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:43.906127 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:43.907122 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:43.907126 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:43.908381 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:43.908395 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:43.909159 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:43.909167 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:43.909376 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:43.910022 (+ 646us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:08:43.910025 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:08:43.910068 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:08:43.910141 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:08:43.910481 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:08:43.910491 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:08:43.910498 (+ 7us) client_negotiation.cc:769] Sending connection context
0808 04:08:43.910667 (+ 169us) client_negotiation.cc:240] Negotiation successful
0808 04:08:43.911067 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":293,"threads_started":1}
I20260808 04:08:43.912333 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161826605655-5452-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161826605655-5452-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161826605655-5452-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.2:0
--local_ip_for_outbound_sockets=127.5.83.2
--webserver_interface=127.5.83.2
--webserver_port=0
--tserver_master_addrs=127.5.83.62:42833
--builtin_ntp_servers=127.5.83.20:36023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 04:08:43.913686 31723 heartbeater.cc:347] Connected to a master server at 127.5.83.62:42833
I20260808 04:08:43.913954 31723 heartbeater.cc:464] Registering TS with master...
I20260808 04:08:43.914659 31723 heartbeater.cc:511] Master 127.5.83.62:42833 requested a full tablet report, sending...
I20260808 04:08:43.916585 31455 ts_manager.cc:194] Registered new tserver with Master: c79d91a0da0e4ed1b8418b97a8e5802a (127.5.83.1:33765)
I20260808 04:08:43.918020 31455 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.83.1:53613
W20260808 04:08:44.237674 31727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:08:44.237924 31727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:08:44.238019 31727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:08:44.244928 31727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:08:44.245013 31727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:08:44.245123 31727 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.2
I20260808 04:08:44.253880 31727 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:36023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161826605655-5452-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161826605655-5452-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.83.2
--webserver_port=0
--tserver_master_addrs=127.5.83.62:42833
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.2
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161826605655-5452-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:08:44.255056 31727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:08:44.256771 31727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:08:44.269054 31736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:08:44.269441 31734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:08:44.269501 31739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:08:44.269954 31727 server_base.cc:1034] running on GCE node
I20260808 04:08:44.270552 31727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:08:44.271386 31727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:08:44.272593 31727 hybrid_clock.cc:648] HybridClock initialized: now 1786162124272548 us; error 40 us; skew 500 ppm
I20260808 04:08:44.272867 31727 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 04:08:44.274924 31727 webserver.cc:460] Webserver started at http://127.5.83.2:42913/ using document root <none> and password file <none>
I20260808 04:08:44.275552 31727 fs_manager.cc:359] Metadata directory not provided
I20260808 04:08:44.275653 31727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:08:44.275971 31727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:08:44.278606 31727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161826605655-5452-0/minicluster-data/ts-1/data/instance:
uuid: "b8ecd9e070a5486f89f0a1e92c524318"
format_stamp: "Formatted at 2026-08-08 04:08:44 on dist-test-slave-1l3l"
server_key: "aee876daa3bb5a5b34f1a3972ae166f9"
server_key_iv: "9ee7499fbaaca36110743e492c7b6040"
server_key_version: "encryptionkey@0"
I20260808 04:08:44.279321 31727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161826605655-5452-0/minicluster-data/ts-1/wal/instance:
uuid: "b8ecd9e070a5486f89f0a1e92c524318"
format_stamp: "Formatted at 2026-08-08 04:08:44 on dist-test-slave-1l3l"
server_key: "aee876daa3bb5a5b34f1a3972ae166f9"
server_key_iv: "9ee7499fbaaca36110743e492c7b6040"
server_key_version: "encryptionkey@0"
I20260808 04:08:44.284011 31727 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260808 04:08:44.292716 31752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:08:44.294137 31727 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.003s sys 0.004s
I20260808 04:08:44.294296 31727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161826605655-5452-0/minicluster-data/ts-1/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161826605655-5452-0/minicluster-data/ts-1/wal
uuid: "b8ecd9e070a5486f89f0a1e92c524318"
format_stamp: "Formatted at 2026-08-08 04:08:44 on dist-test-slave-1l3l"
server_key: "aee876daa3bb5a5b34f1a3972ae166f9"
server_key_iv: "9ee7499fbaaca36110743e492c7b6040"
server_key_version: "encryptionkey@0"
I20260808 04:08:44.294567 31727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161826605655-5452-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161826605655-5452-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161826605655-5452-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:08:44.308892 31727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:08:44.309657 31727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:08:44.309898 31727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:08:44.310783 31727 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:08:44.312273 31727 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:08:44.312372 31727 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:08:44.312453 31727 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:08:44.312505 31727 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:08:44.341656 31727 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.2:33041
I20260808 04:08:44.341712 31911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.2:33041 every 8 connection(s)
I20260808 04:08:44.342867 31727 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
I20260808 04:08:44.343271 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 31727
I20260808 04:08:44.343431 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161826605655-5452-0/minicluster-data/ts-1/wal/instance
I20260808 04:08:44.344046 5452 external_mini_cluster.cc:1428] Setting key 84c25cf0899170711edb89bd00cb4cd3
I20260808 04:08:44.347039 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161826605655-5452-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161826605655-5452-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161826605655-5452-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.3:0
--local_ip_for_outbound_sockets=127.5.83.3
--webserver_interface=127.5.83.3
--webserver_port=0
--tserver_master_addrs=127.5.83.62:42833
--builtin_ntp_servers=127.5.83.20:36023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 04:08:44.357134 31725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:44.345414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.2:44887 (local address 127.5.83.62:42833)
0808 04:08:44.345618 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:44.345624 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:44.347189 (+ 1565us) server_negotiation.cc:407] Connection header received
0808 04:08:44.349546 (+ 2357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:44.349552 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:44.349635 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:44.349733 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:08:44.350969 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:44.352077 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:44.354082 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:44.354700 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:44.354816 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:44.356338 (+ 1522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:08:44.356345 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:08:44.356350 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:08:44.356370 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:08:44.356409 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:08:44.356423 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:08:44.356510 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:44.356734 (+ 224us) server_negotiation.cc:299] Negotiation successful
0808 04:08:44.356828 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 04:08:44.358177 31916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:44.345980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:42833 (local address 127.5.83.2:44887)
0808 04:08:44.347018 (+ 1038us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:44.347067 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0808 04:08:44.349210 (+ 2143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:08:44.349966 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:08:44.349975 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:08:44.350079 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:08:44.350791 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:44.350813 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:44.352255 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:44.352260 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:44.353934 (+ 1674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:44.353948 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:44.354920 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:44.354927 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:44.355187 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:44.356010 (+ 823us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:08:44.356013 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:08:44.356061 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:08:44.356145 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:08:44.356525 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:08:44.356535 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:08:44.356542 (+ 7us) client_negotiation.cc:769] Sending connection context
0808 04:08:44.356711 (+ 169us) client_negotiation.cc:240] Negotiation successful
0808 04:08:44.357043 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":317,"threads_started":1}
I20260808 04:08:44.359503 31912 heartbeater.cc:347] Connected to a master server at 127.5.83.62:42833
I20260808 04:08:44.359809 31912 heartbeater.cc:464] Registering TS with master...
I20260808 04:08:44.360594 31912 heartbeater.cc:511] Master 127.5.83.62:42833 requested a full tablet report, sending...
I20260808 04:08:44.361891 31455 ts_manager.cc:194] Registered new tserver with Master: b8ecd9e070a5486f89f0a1e92c524318 (127.5.83.2:33041)
I20260808 04:08:44.362682 31455 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.83.2:44887
W20260808 04:08:44.670131 31919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:08:44.670382 31919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:08:44.670475 31919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:08:44.677431 31919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:08:44.677520 31919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:08:44.677626 31919 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.3
I20260808 04:08:44.686120 31919 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:36023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161826605655-5452-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161826605655-5452-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.83.3
--webserver_port=0
--tserver_master_addrs=127.5.83.62:42833
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.3
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161826605655-5452-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:08:44.687227 31919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:08:44.688871 31919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:08:44.701386 31930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:08:44.701327 31933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:08:44.701361 31931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:08:44.702541 31919 server_base.cc:1034] running on GCE node
I20260808 04:08:44.703070 31919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:08:44.703935 31919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:08:44.705128 31919 hybrid_clock.cc:648] HybridClock initialized: now 1786162124705082 us; error 43 us; skew 500 ppm
I20260808 04:08:44.705416 31919 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 04:08:44.707520 31919 webserver.cc:460] Webserver started at http://127.5.83.3:45073/ using document root <none> and password file <none>
I20260808 04:08:44.708168 31919 fs_manager.cc:359] Metadata directory not provided
I20260808 04:08:44.708268 31919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:08:44.708572 31919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:08:44.711170 31919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161826605655-5452-0/minicluster-data/ts-2/data/instance:
uuid: "387d7d4d48124399b1ad4dd2ab7d4589"
format_stamp: "Formatted at 2026-08-08 04:08:44 on dist-test-slave-1l3l"
server_key: "90dddfd76294c0a22147b452266b5118"
server_key_iv: "b8f97245f9f0857cf53f3b6d5587d6d5"
server_key_version: "encryptionkey@0"
I20260808 04:08:44.711956 31919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161826605655-5452-0/minicluster-data/ts-2/wal/instance:
uuid: "387d7d4d48124399b1ad4dd2ab7d4589"
format_stamp: "Formatted at 2026-08-08 04:08:44 on dist-test-slave-1l3l"
server_key: "90dddfd76294c0a22147b452266b5118"
server_key_iv: "b8f97245f9f0857cf53f3b6d5587d6d5"
server_key_version: "encryptionkey@0"
I20260808 04:08:44.716543 31919 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260808 04:08:44.719563 31942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:08:44.720731 31919 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 04:08:44.720872 31919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161826605655-5452-0/minicluster-data/ts-2/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161826605655-5452-0/minicluster-data/ts-2/wal
uuid: "387d7d4d48124399b1ad4dd2ab7d4589"
format_stamp: "Formatted at 2026-08-08 04:08:44 on dist-test-slave-1l3l"
server_key: "90dddfd76294c0a22147b452266b5118"
server_key_iv: "b8f97245f9f0857cf53f3b6d5587d6d5"
server_key_version: "encryptionkey@0"
I20260808 04:08:44.721055 31919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161826605655-5452-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161826605655-5452-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161826605655-5452-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:08:44.732744 31919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:08:44.733433 31919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:08:44.733675 31919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:08:44.734534 31919 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:08:44.735916 31919 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:08:44.736012 31919 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:08:44.736120 31919 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:08:44.736191 31919 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:08:44.765590 31919 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.3:39777
I20260808 04:08:44.765641 32110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.3:39777 every 8 connection(s)
I20260808 04:08:44.766784 31919 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
I20260808 04:08:44.767973 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 31919
I20260808 04:08:44.768146 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161826605655-5452-0/minicluster-data/ts-2/wal/instance
I20260808 04:08:44.768716 5452 external_mini_cluster.cc:1428] Setting key baf7f5fd48beea880b6d9e780c417b32
I20260808 04:08:44.778326 31725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:44.769461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.3:56043 (local address 127.5.83.62:42833)
0808 04:08:44.769653 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:44.769659 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:44.771121 (+ 1462us) server_negotiation.cc:407] Connection header received
0808 04:08:44.772860 (+ 1739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:44.772864 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:44.772930 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:44.773007 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:08:44.773871 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:44.774601 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:44.776127 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:44.776541 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:44.776634 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:44.777727 (+ 1093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:08:44.777732 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:08:44.777735 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:08:44.777751 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:08:44.777781 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:08:44.777792 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:08:44.777856 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:44.778015 (+ 159us) server_negotiation.cc:299] Negotiation successful
0808 04:08:44.778088 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260808 04:08:44.779177 32114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:44.769966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:42833 (local address 127.5.83.3:56043)
0808 04:08:44.770970 (+ 1004us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:44.771017 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0808 04:08:44.772649 (+ 1632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:08:44.773151 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:08:44.773159 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:08:44.773241 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:08:44.773743 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:44.773759 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:44.774712 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:44.774715 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:44.776011 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:44.776022 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:44.776701 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:44.776705 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:44.776906 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:44.777466 (+ 560us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:08:44.777469 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:08:44.777508 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:08:44.777574 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:08:44.777861 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:08:44.777867 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:08:44.777872 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 04:08:44.778007 (+ 135us) client_negotiation.cc:240] Negotiation successful
0808 04:08:44.778247 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":295,"threads_started":1}
I20260808 04:08:44.780360 32111 heartbeater.cc:347] Connected to a master server at 127.5.83.62:42833
I20260808 04:08:44.780606 32111 heartbeater.cc:464] Registering TS with master...
I20260808 04:08:44.781221 32111 heartbeater.cc:511] Master 127.5.83.62:42833 requested a full tablet report, sending...
I20260808 04:08:44.782572 31454 ts_manager.cc:194] Registered new tserver with Master: 387d7d4d48124399b1ad4dd2ab7d4589 (127.5.83.3:39777)
I20260808 04:08:44.783507 5452 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:08:44.783560 31454 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.83.3:56043
I20260808 04:08:44.792871 31725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:44.787492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52202 (local address 127.5.83.62:42833)
0808 04:08:44.787646 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:44.787650 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:44.788124 (+ 474us) server_negotiation.cc:407] Connection header received
0808 04:08:44.788264 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:44.788269 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:44.788354 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:44.788469 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:08:44.788839 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:44.789482 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:44.790840 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:44.791256 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:44.791339 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:44.792340 (+ 1001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:08:44.792345 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:08:44.792348 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:08:44.792364 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:08:44.792395 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:08:44.792414 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:08:44.792479 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:44.792584 (+ 105us) server_negotiation.cc:299] Negotiation successful
0808 04:08:44.792655 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 04:08:44.796653 31454 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52202:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:08:44.803499 31454 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:08:44.839341 32132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:44.815008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.3:39777 (local address 127.0.0.1:56356)
0808 04:08:44.831991 (+ 16983us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:44.832007 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:08:44.832207 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:08:44.832952 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:08:44.832956 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:08:44.832978 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:08:44.833332 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:44.833341 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:44.834622 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:44.834627 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:44.836948 (+ 2321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:44.836961 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:44.838506 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:44.838512 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:44.838707 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:44.838760 (+ 53us) client_negotiation.cc:769] Sending connection context
0808 04:08:44.838880 (+ 120us) client_negotiation.cc:240] Negotiation successful
0808 04:08:44.839003 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16863,"mutex_wait_us":403,"thread_start_us":16020,"threads_started":1}
I20260808 04:08:44.840003 32133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:44.814929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.2:33041 (local address 127.0.0.1:40912)
0808 04:08:44.831295 (+ 16366us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:44.831314 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 04:08:44.831445 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:08:44.832475 (+ 1030us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:08:44.832479 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:08:44.832506 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:08:44.832965 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:44.832977 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:44.834873 (+ 1896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:44.834878 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:44.837476 (+ 2598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:44.837491 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:44.839230 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:44.839236 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:44.839411 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:44.839446 (+ 35us) client_negotiation.cc:769] Sending connection context
0808 04:08:44.839521 (+ 75us) client_negotiation.cc:240] Negotiation successful
0808 04:08:44.839641 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16243,"mutex_wait_us":29,"thread_start_us":16797,"threads_started":1}
I20260808 04:08:44.840147 32137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:44.816269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56356 (local address 127.5.83.3:39777)
0808 04:08:44.823379 (+ 7110us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:44.823389 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:44.832108 (+ 8719us) server_negotiation.cc:407] Connection header received
0808 04:08:44.832418 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:44.832426 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:44.832630 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:44.832796 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:08:44.833482 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:44.834489 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:44.837143 (+ 2654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:44.838285 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:44.838458 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:44.838731 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:44.839607 (+ 876us) server_negotiation.cc:299] Negotiation successful
0808 04:08:44.839798 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6914,"thread_start_us":6734,"threads_started":1}
I20260808 04:08:44.841715 32135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:44.815763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40912 (local address 127.5.83.2:33041)
0808 04:08:44.830869 (+ 15106us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:44.830878 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:44.831576 (+ 698us) server_negotiation.cc:407] Connection header received
0808 04:08:44.831693 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:44.831732 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:44.831988 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:44.832181 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:08:44.833155 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:44.834701 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:44.837693 (+ 2992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:44.839036 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:44.839175 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:44.839785 (+ 610us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:44.839867 (+ 82us) server_negotiation.cc:299] Negotiation successful
0808 04:08:44.841344 (+ 1477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14951,"thread_start_us":14797,"threads_started":1}
I20260808 04:08:44.842260 32127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:44.814591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:33765 (local address 127.0.0.1:51774)
0808 04:08:44.830074 (+ 15483us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:44.830121 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0808 04:08:44.830300 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:08:44.832223 (+ 1923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:08:44.832230 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:08:44.832269 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:08:44.832722 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:44.832734 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:44.834894 (+ 2160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:44.834908 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:44.837478 (+ 2570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:44.837491 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:44.841314 (+ 3823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:44.841321 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:44.841592 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:44.841624 (+ 32us) client_negotiation.cc:769] Sending connection context
0808 04:08:44.841718 (+ 94us) client_negotiation.cc:240] Negotiation successful
0808 04:08:44.841940 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15195,"mutex_wait_us":200,"thread_start_us":15037,"threads_started":1}
I20260808 04:08:44.842265 32131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:44.814707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51774 (local address 127.5.83.1:33765)
0808 04:08:44.831629 (+ 16922us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:44.831636 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:44.831659 (+ 23us) server_negotiation.cc:407] Connection header received
0808 04:08:44.831766 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:44.831772 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:44.831979 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:44.832885 (+ 906us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:08:44.833116 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:44.834758 (+ 1642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:44.839812 (+ 5054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:44.841114 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:44.841247 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:44.841469 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:44.841779 (+ 310us) server_negotiation.cc:299] Negotiation successful
0808 04:08:44.841938 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16763,"thread_start_us":16661,"threads_started":1}
I20260808 04:08:44.843977 32013 tablet_service.cc:1467] Processing CreateTablet for tablet 03d2c0a3a9304e3b99a2c4d9bc9ebd52 (DEFAULT_TABLE table=test-table [id=36d6cc5a5d334436b8f7857f7e1f58da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:08:44.845463 32013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03d2c0a3a9304e3b99a2c4d9bc9ebd52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:08:44.845014 31823 tablet_service.cc:1467] Processing CreateTablet for tablet 03d2c0a3a9304e3b99a2c4d9bc9ebd52 (DEFAULT_TABLE table=test-table [id=36d6cc5a5d334436b8f7857f7e1f58da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:08:44.846139 31823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03d2c0a3a9304e3b99a2c4d9bc9ebd52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:08:44.846990 31630 tablet_service.cc:1467] Processing CreateTablet for tablet 03d2c0a3a9304e3b99a2c4d9bc9ebd52 (DEFAULT_TABLE table=test-table [id=36d6cc5a5d334436b8f7857f7e1f58da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:08:44.848272 31630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03d2c0a3a9304e3b99a2c4d9bc9ebd52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:08:44.856932 32138 tablet_bootstrap.cc:492] T 03d2c0a3a9304e3b99a2c4d9bc9ebd52 P b8ecd9e070a5486f89f0a1e92c524318: Bootstrap starting.
I20260808 04:08:44.859280 32139 tablet_bootstrap.cc:492] T 03d2c0a3a9304e3b99a2c4d9bc9ebd52 P 387d7d4d48124399b1ad4dd2ab7d4589: Bootstrap starting.
I20260808 04:08:44.860689 32138 tablet_bootstrap.cc:654] T 03d2c0a3a9304e3b99a2c4d9bc9ebd52 P b8ecd9e070a5486f89f0a1e92c524318: Neither blocks nor log segments found. Creating new log.
I20260808 04:08:44.861701 32140 tablet_bootstrap.cc:492] T 03d2c0a3a9304e3b99a2c4d9bc9ebd52 P c79d91a0da0e4ed1b8418b97a8e5802a: Bootstrap starting.
I20260808 04:08:44.861917 32138 log.cc:824] T 03d2c0a3a9304e3b99a2c4d9bc9ebd52 P b8ecd9e070a5486f89f0a1e92c524318: Log is configured to *not* fsync() on all Append() calls
I20260808 04:08:44.862643 32139 tablet_bootstrap.cc:654] T 03d2c0a3a9304e3b99a2c4d9bc9ebd52 P 387d7d4d48124399b1ad4dd2ab7d4589: Neither blocks nor log segments found. Creating new log.
I20260808 04:08:44.863917 32139 log.cc:824] T 03d2c0a3a9304e3b99a2c4d9bc9ebd52 P 387d7d4d48124399b1ad4dd2ab7d4589: Log is configured to *not* fsync() on all Append() calls
I20260808 04:08:44.864902 32140 tablet_bootstrap.cc:654] T 03d2c0a3a9304e3b99a2c4d9bc9ebd52 P c79d91a0da0e4ed1b8418b97a8e5802a: Neither blocks nor log segments found. Creating new log.
I20260808 04:08:44.865746 32140 log.cc:824] T 03d2c0a3a9304e3b99a2c4d9bc9ebd52 P c79d91a0da0e4ed1b8418b97a8e5802a: Log is configured to *not* fsync() on all Append() calls
I20260808 04:08:44.866516 32138 tablet_bootstrap.cc:492] T 03d2c0a3a9304e3b99a2c4d9bc9ebd52 P b8ecd9e070a5486f89f0a1e92c524318: No bootstrap required, opened a new log
I20260808 04:08:44.866796 32138 ts_tablet_manager.cc:1397] T 03d2c0a3a9304e3b99a2c4d9bc9ebd52 P b8ecd9e070a5486f89f0a1e92c524318: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260808 04:08:44.868052 32140 tablet_bootstrap.cc:492] T 03d2c0a3a9304e3b99a2c4d9bc9ebd52 P c79d91a0da0e4ed1b8418b97a8e5802a: No bootstrap required, opened a new log
I20260808 04:08:44.868270 32140 ts_tablet_manager.cc:1397] T 03d2c0a3a9304e3b99a2c4d9bc9ebd52 P c79d91a0da0e4ed1b8418b97a8e5802a: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260808 04:08:44.872951 32139 tablet_bootstrap.cc:492] T 03d2c0a3a9304e3b99a2c4d9bc9ebd52 P 387d7d4d48124399b1ad4dd2ab7d4589: No bootstrap required, opened a new log
I20260808 04:08:44.873207 32139 ts_tablet_manager.cc:1397] T 03d2c0a3a9304e3b99a2c4d9bc9ebd52 P 387d7d4d48124399b1ad4dd2ab7d4589: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.001s
I20260808 04:08:44.875382 32138 raft_consensus.cc:348] T 03d2c0a3a9304e3b99a2c4d9bc9ebd52 P b8ecd9e070a5486f89f0a1e92c524318 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c79d91a0da0e4ed1b8418b97a8e5802a" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 33765 } } peers { permanent_uuid: "b8ecd9e070a5486f89f0a1e92c524318" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 33041 } } peers { permanent_uuid: "387d7d4d48124399b1ad4dd2ab7d4589" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 39777 } }
I20260808 04:08:44.875756 32138 raft_consensus.cc:374] T 03d2c0a3a9304e3b99a2c4d9bc9ebd52 P b8ecd9e070a5486f89f0a1e92c524318 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:08:44.875854 32138 raft_consensus.cc:729] T 03d2c0a3a9304e3b99a2c4d9bc9ebd52 P b8ecd9e070a5486f89f0a1e92c524318 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8ecd9e070a5486f89f0a1e92c524318, State: Initialized, Role: FOLLOWER
I20260808 04:08:44.876518 32138 consensus_queue.cc:260] T 03d2c0a3a9304e3b99a2c4d9bc9ebd52 P b8ecd9e070a5486f89f0a1e92c524318 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c79d91a0da0e4ed1b8418b97a8e5802a" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 33765 } } peers { permanent_uuid: "b8ecd9e070a5486f89f0a1e92c524318" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 33041 } } peers { permanent_uuid: "387d7d4d48124399b1ad4dd2ab7d4589" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 39777 } }
I20260808 04:08:44.877240 32140 raft_consensus.cc:348] T 03d2c0a3a9304e3b99a2c4d9bc9ebd52 P c79d91a0da0e4ed1b8418b97a8e5802a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c79d91a0da0e4ed1b8418b97a8e5802a" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 33765 } } peers { permanent_uuid: "b8ecd9e070a5486f89f0a1e92c524318" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 33041 } } peers { permanent_uuid: "387d7d4d48124399b1ad4dd2ab7d4589" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 39777 } }
I20260808 04:08:44.877527 31912 heartbeater.cc:503] Master 127.5.83.62:42833 was elected leader, sending a full tablet report...
I20260808 04:08:44.877646 32140 raft_consensus.cc:374] T 03d2c0a3a9304e3b99a2c4d9bc9ebd52 P c79d91a0da0e4ed1b8418b97a8e5802a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:08:44.877729 32140 raft_consensus.cc:729] T 03d2c0a3a9304e3b99a2c4d9bc9ebd52 P c79d91a0da0e4ed1b8418b97a8e5802a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c79d91a0da0e4ed1b8418b97a8e5802a, State: Initialized, Role: FOLLOWER
I20260808 04:08:44.878139 32138 ts_tablet_manager.cc:1428] T 03d2c0a3a9304e3b99a2c4d9bc9ebd52 P b8ecd9e070a5486f89f0a1e92c524318: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260808 04:08:44.878351 32140 consensus_queue.cc:260] T 03d2c0a3a9304e3b99a2c4d9bc9ebd52 P c79d91a0da0e4ed1b8418b97a8e5802a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c79d91a0da0e4ed1b8418b97a8e5802a" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 33765 } } peers { permanent_uuid: "b8ecd9e070a5486f89f0a1e92c524318" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 33041 } } peers { permanent_uuid: "387d7d4d48124399b1ad4dd2ab7d4589" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 39777 } }
I20260808 04:08:44.879940 32140 ts_tablet_manager.cc:1428] T 03d2c0a3a9304e3b99a2c4d9bc9ebd52 P c79d91a0da0e4ed1b8418b97a8e5802a: Time spent starting tablet: real 0.012s user 0.010s sys 0.002s
I20260808 04:08:44.880255 31723 heartbeater.cc:503] Master 127.5.83.62:42833 was elected leader, sending a full tablet report...
I20260808 04:08:44.882477 32139 raft_consensus.cc:348] T 03d2c0a3a9304e3b99a2c4d9bc9ebd52 P 387d7d4d48124399b1ad4dd2ab7d4589 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c79d91a0da0e4ed1b8418b97a8e5802a" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 33765 } } peers { permanent_uuid: "b8ecd9e070a5486f89f0a1e92c524318" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 33041 } } peers { permanent_uuid: "387d7d4d48124399b1ad4dd2ab7d4589" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 39777 } }
I20260808 04:08:44.882889 32139 raft_consensus.cc:374] T 03d2c0a3a9304e3b99a2c4d9bc9ebd52 P 387d7d4d48124399b1ad4dd2ab7d4589 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:08:44.883018 32139 raft_consensus.cc:729] T 03d2c0a3a9304e3b99a2c4d9bc9ebd52 P 387d7d4d48124399b1ad4dd2ab7d4589 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 387d7d4d48124399b1ad4dd2ab7d4589, State: Initialized, Role: FOLLOWER
I20260808 04:08:44.883566 32139 consensus_queue.cc:260] T 03d2c0a3a9304e3b99a2c4d9bc9ebd52 P 387d7d4d48124399b1ad4dd2ab7d4589 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c79d91a0da0e4ed1b8418b97a8e5802a" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 33765 } } peers { permanent_uuid: "b8ecd9e070a5486f89f0a1e92c524318" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 33041 } } peers { permanent_uuid: "387d7d4d48124399b1ad4dd2ab7d4589" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 39777 } }
I20260808 04:08:44.884502 32111 heartbeater.cc:503] Master 127.5.83.62:42833 was elected leader, sending a full tablet report...
I20260808 04:08:44.885015 32139 ts_tablet_manager.cc:1428] T 03d2c0a3a9304e3b99a2c4d9bc9ebd52 P 387d7d4d48124399b1ad4dd2ab7d4589: Time spent starting tablet: real 0.012s user 0.006s sys 0.006s
W20260808 04:08:44.901439 31724 tablet.cc:2369] T 03d2c0a3a9304e3b99a2c4d9bc9ebd52 P c79d91a0da0e4ed1b8418b97a8e5802a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 04:08:45.018388 32112 tablet.cc:2369] T 03d2c0a3a9304e3b99a2c4d9bc9ebd52 P 387d7d4d48124399b1ad4dd2ab7d4589: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:08:45.079226 32150 raft_consensus.cc:482] T 03d2c0a3a9304e3b99a2c4d9bc9ebd52 P 387d7d4d48124399b1ad4dd2ab7d4589 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:08:45.079519 32150 raft_consensus.cc:504] T 03d2c0a3a9304e3b99a2c4d9bc9ebd52 P 387d7d4d48124399b1ad4dd2ab7d4589 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c79d91a0da0e4ed1b8418b97a8e5802a" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 33765 } } peers { permanent_uuid: "b8ecd9e070a5486f89f0a1e92c524318" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 33041 } } peers { permanent_uuid: "387d7d4d48124399b1ad4dd2ab7d4589" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 39777 } }
I20260808 04:08:45.081229 32150 leader_election.cc:288] T 03d2c0a3a9304e3b99a2c4d9bc9ebd52 P 387d7d4d48124399b1ad4dd2ab7d4589 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c79d91a0da0e4ed1b8418b97a8e5802a (127.5.83.1:33765), b8ecd9e070a5486f89f0a1e92c524318 (127.5.83.2:33041)
I20260808 04:08:45.087392 32131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:45.081649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.3:48715 (local address 127.5.83.1:33765)
0808 04:08:45.081827 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:45.081832 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:45.081853 (+ 21us) server_negotiation.cc:407] Connection header received
0808 04:08:45.082031 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:45.082037 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:45.082134 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:45.082228 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:08:45.082629 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:45.083531 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:45.085540 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:45.086632 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:45.086728 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:45.086793 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:45.087113 (+ 320us) server_negotiation.cc:299] Negotiation successful
0808 04:08:45.087197 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 04:08:45.087514 32114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:45.081469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:33765 (local address 127.5.83.3:48715)
0808 04:08:45.081677 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:45.081698 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 04:08:45.081834 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:08:45.082280 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:08:45.082284 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:08:45.082310 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:08:45.082469 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:45.082477 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:45.083666 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:45.083671 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:45.085371 (+ 1700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:45.085384 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:45.086783 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:45.086789 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:45.086966 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:45.087003 (+ 37us) client_negotiation.cc:769] Sending connection context
0808 04:08:45.087098 (+ 95us) client_negotiation.cc:240] Negotiation successful
0808 04:08:45.087232 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260808 04:08:45.088330 32135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:45.081864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.3:41007 (local address 127.5.83.2:33041)
0808 04:08:45.082087 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:45.082093 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:45.082476 (+ 383us) server_negotiation.cc:407] Connection header received
0808 04:08:45.082660 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:45.082664 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:45.082721 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:45.082788 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:08:45.083346 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:45.084311 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:45.086341 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:45.087393 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:45.087521 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:45.087612 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:45.088066 (+ 454us) server_negotiation.cc:299] Negotiation successful
0808 04:08:45.088165 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260808 04:08:45.088379 32153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:45.081581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.2:33041 (local address 127.5.83.3:41007)
0808 04:08:45.082364 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:45.082388 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0808 04:08:45.082512 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:08:45.082824 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:08:45.082829 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:08:45.082854 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:08:45.083181 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:45.083192 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:45.084458 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:45.084461 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:45.086135 (+ 1674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:45.086150 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:45.087698 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:45.087723 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:45.087917 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:45.087940 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 04:08:45.088010 (+ 70us) client_negotiation.cc:240] Negotiation successful
0808 04:08:45.088103 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"thread_start_us":434,"threads_started":1}
I20260808 04:08:45.088614 31656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03d2c0a3a9304e3b99a2c4d9bc9ebd52" candidate_uuid: "387d7d4d48124399b1ad4dd2ab7d4589" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c79d91a0da0e4ed1b8418b97a8e5802a" is_pre_election: true
I20260808 04:08:45.089154 31656 raft_consensus.cc:2454] T 03d2c0a3a9304e3b99a2c4d9bc9ebd52 P c79d91a0da0e4ed1b8418b97a8e5802a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 387d7d4d48124399b1ad4dd2ab7d4589 in term 0.
I20260808 04:08:45.089335 31848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03d2c0a3a9304e3b99a2c4d9bc9ebd52" candidate_uuid: "387d7d4d48124399b1ad4dd2ab7d4589" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8ecd9e070a5486f89f0a1e92c524318" is_pre_election: true
I20260808 04:08:45.089805 31848 raft_consensus.cc:2454] T 03d2c0a3a9304e3b99a2c4d9bc9ebd52 P b8ecd9e070a5486f89f0a1e92c524318 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 387d7d4d48124399b1ad4dd2ab7d4589 in term 0.
I20260808 04:08:45.089833 31947 leader_election.cc:302] T 03d2c0a3a9304e3b99a2c4d9bc9ebd52 P 387d7d4d48124399b1ad4dd2ab7d4589 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 387d7d4d48124399b1ad4dd2ab7d4589, c79d91a0da0e4ed1b8418b97a8e5802a; no voters:
I20260808 04:08:45.090174 32150 raft_consensus.cc:2781] T 03d2c0a3a9304e3b99a2c4d9bc9ebd52 P 387d7d4d48124399b1ad4dd2ab7d4589 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:08:45.090325 32150 raft_consensus.cc:482] T 03d2c0a3a9304e3b99a2c4d9bc9ebd52 P 387d7d4d48124399b1ad4dd2ab7d4589 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:08:45.090446 32150 raft_consensus.cc:3037] T 03d2c0a3a9304e3b99a2c4d9bc9ebd52 P 387d7d4d48124399b1ad4dd2ab7d4589 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:08:45.092736 32150 raft_consensus.cc:504] T 03d2c0a3a9304e3b99a2c4d9bc9ebd52 P 387d7d4d48124399b1ad4dd2ab7d4589 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c79d91a0da0e4ed1b8418b97a8e5802a" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 33765 } } peers { permanent_uuid: "b8ecd9e070a5486f89f0a1e92c524318" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 33041 } } peers { permanent_uuid: "387d7d4d48124399b1ad4dd2ab7d4589" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 39777 } }
I20260808 04:08:45.093243 32150 leader_election.cc:288] T 03d2c0a3a9304e3b99a2c4d9bc9ebd52 P 387d7d4d48124399b1ad4dd2ab7d4589 [CANDIDATE]: Term 1 election: Requested vote from peers c79d91a0da0e4ed1b8418b97a8e5802a (127.5.83.1:33765), b8ecd9e070a5486f89f0a1e92c524318 (127.5.83.2:33041)
I20260808 04:08:45.093612 31656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03d2c0a3a9304e3b99a2c4d9bc9ebd52" candidate_uuid: "387d7d4d48124399b1ad4dd2ab7d4589" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c79d91a0da0e4ed1b8418b97a8e5802a"
I20260808 04:08:45.093708 31848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03d2c0a3a9304e3b99a2c4d9bc9ebd52" candidate_uuid: "387d7d4d48124399b1ad4dd2ab7d4589" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8ecd9e070a5486f89f0a1e92c524318"
I20260808 04:08:45.093776 31656 raft_consensus.cc:3037] T 03d2c0a3a9304e3b99a2c4d9bc9ebd52 P c79d91a0da0e4ed1b8418b97a8e5802a [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:08:45.093894 31848 raft_consensus.cc:3037] T 03d2c0a3a9304e3b99a2c4d9bc9ebd52 P b8ecd9e070a5486f89f0a1e92c524318 [term 0 FOLLOWER]: Advancing to term 1
W20260808 04:08:45.095376 31915 tablet.cc:2369] T 03d2c0a3a9304e3b99a2c4d9bc9ebd52 P b8ecd9e070a5486f89f0a1e92c524318: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:08:45.096377 31656 raft_consensus.cc:2454] T 03d2c0a3a9304e3b99a2c4d9bc9ebd52 P c79d91a0da0e4ed1b8418b97a8e5802a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 387d7d4d48124399b1ad4dd2ab7d4589 in term 1.
I20260808 04:08:45.096397 31848 raft_consensus.cc:2454] T 03d2c0a3a9304e3b99a2c4d9bc9ebd52 P b8ecd9e070a5486f89f0a1e92c524318 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 387d7d4d48124399b1ad4dd2ab7d4589 in term 1.
I20260808 04:08:45.096714 31947 leader_election.cc:302] T 03d2c0a3a9304e3b99a2c4d9bc9ebd52 P 387d7d4d48124399b1ad4dd2ab7d4589 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 387d7d4d48124399b1ad4dd2ab7d4589, c79d91a0da0e4ed1b8418b97a8e5802a; no voters:
I20260808 04:08:45.096976 32150 raft_consensus.cc:2781] T 03d2c0a3a9304e3b99a2c4d9bc9ebd52 P 387d7d4d48124399b1ad4dd2ab7d4589 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:08:45.097447 32150 raft_consensus.cc:686] T 03d2c0a3a9304e3b99a2c4d9bc9ebd52 P 387d7d4d48124399b1ad4dd2ab7d4589 [term 1 LEADER]: Becoming Leader. State: Replica: 387d7d4d48124399b1ad4dd2ab7d4589, State: Running, Role: LEADER
I20260808 04:08:45.097941 32150 consensus_queue.cc:237] T 03d2c0a3a9304e3b99a2c4d9bc9ebd52 P 387d7d4d48124399b1ad4dd2ab7d4589 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c79d91a0da0e4ed1b8418b97a8e5802a" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 33765 } } peers { permanent_uuid: "b8ecd9e070a5486f89f0a1e92c524318" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 33041 } } peers { permanent_uuid: "387d7d4d48124399b1ad4dd2ab7d4589" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 39777 } }
I20260808 04:08:45.102492 31455 catalog_manager.cc:5495] T 03d2c0a3a9304e3b99a2c4d9bc9ebd52 P 387d7d4d48124399b1ad4dd2ab7d4589 reported cstate change: term changed from 0 to 1, leader changed from <none> to 387d7d4d48124399b1ad4dd2ab7d4589 (127.5.83.3). New cstate: current_term: 1 leader_uuid: "387d7d4d48124399b1ad4dd2ab7d4589" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c79d91a0da0e4ed1b8418b97a8e5802a" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 33765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8ecd9e070a5486f89f0a1e92c524318" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 33041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "387d7d4d48124399b1ad4dd2ab7d4589" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 39777 } health_report { overall_health: HEALTHY } } }
I20260808 04:08:45.134976 32137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:45.129556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56360 (local address 127.5.83.3:39777)
0808 04:08:45.129759 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:45.129765 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:45.129785 (+ 20us) server_negotiation.cc:407] Connection header received
0808 04:08:45.129944 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:45.129948 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:45.130024 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:45.130150 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:08:45.130632 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:45.131438 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:45.132891 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:45.133305 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:45.133422 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:45.133918 (+ 496us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:08:45.134418 (+ 500us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:08:45.134551 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:45.134655 (+ 104us) server_negotiation.cc:299] Negotiation successful
0808 04:08:45.134734 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 04:08:45.142637 31848 raft_consensus.cc:1261] T 03d2c0a3a9304e3b99a2c4d9bc9ebd52 P b8ecd9e070a5486f89f0a1e92c524318 [term 1 FOLLOWER]: Refusing update from remote peer 387d7d4d48124399b1ad4dd2ab7d4589: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:08:45.142638 31656 raft_consensus.cc:1261] T 03d2c0a3a9304e3b99a2c4d9bc9ebd52 P c79d91a0da0e4ed1b8418b97a8e5802a [term 1 FOLLOWER]: Refusing update from remote peer 387d7d4d48124399b1ad4dd2ab7d4589: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:08:45.143496 32154 consensus_queue.cc:1035] T 03d2c0a3a9304e3b99a2c4d9bc9ebd52 P 387d7d4d48124399b1ad4dd2ab7d4589 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c79d91a0da0e4ed1b8418b97a8e5802a" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 33765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:08:45.143889 32150 consensus_queue.cc:1035] T 03d2c0a3a9304e3b99a2c4d9bc9ebd52 P 387d7d4d48124399b1ad4dd2ab7d4589 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8ecd9e070a5486f89f0a1e92c524318" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 33041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:08:45.155519 32163 mvcc.cc:205] Tried to move back new op lower bound from 7316120064575209472 to 7316120064404619264. Current Snapshot: MvccSnapshot[applied={T|T < 7316120064575209472}]
I20260808 04:08:45.166749 31454 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:08:45.166990 31454 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:08:45.170547 31454 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eb7da5edad5c436e884c6e3fe1cf80f6: Sending DeleteTablet for 3 replicas of tablet 03d2c0a3a9304e3b99a2c4d9bc9ebd52
I20260808 04:08:45.171626 31823 tablet_service.cc:1514] Processing DeleteTablet for tablet 03d2c0a3a9304e3b99a2c4d9bc9ebd52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:08:45 UTC) from {username='slave'} at 127.0.0.1:40912
I20260808 04:08:45.171648 31630 tablet_service.cc:1514] Processing DeleteTablet for tablet 03d2c0a3a9304e3b99a2c4d9bc9ebd52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:08:45 UTC) from {username='slave'} at 127.0.0.1:51774
I20260808 04:08:45.171877 32013 tablet_service.cc:1514] Processing DeleteTablet for tablet 03d2c0a3a9304e3b99a2c4d9bc9ebd52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:08:45 UTC) from {username='slave'} at 127.0.0.1:56356
I20260808 04:08:45.172976 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 31538
I20260808 04:08:45.186911 32176 tablet_replica.cc:330] T 03d2c0a3a9304e3b99a2c4d9bc9ebd52 P c79d91a0da0e4ed1b8418b97a8e5802a: stopping tablet replica
I20260808 04:08:45.187352 32176 raft_consensus.cc:2229] T 03d2c0a3a9304e3b99a2c4d9bc9ebd52 P c79d91a0da0e4ed1b8418b97a8e5802a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:08:45.187884 32176 raft_consensus.cc:2258] T 03d2c0a3a9304e3b99a2c4d9bc9ebd52 P c79d91a0da0e4ed1b8418b97a8e5802a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:08:45.190073 32176 ts_tablet_manager.cc:1905] T 03d2c0a3a9304e3b99a2c4d9bc9ebd52 P c79d91a0da0e4ed1b8418b97a8e5802a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:08:45.190153 32173 tablet_replica.cc:330] T 03d2c0a3a9304e3b99a2c4d9bc9ebd52 P b8ecd9e070a5486f89f0a1e92c524318: stopping tablet replica
I20260808 04:08:45.190550 32173 raft_consensus.cc:2229] T 03d2c0a3a9304e3b99a2c4d9bc9ebd52 P b8ecd9e070a5486f89f0a1e92c524318 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:08:45.191025 32173 raft_consensus.cc:2258] T 03d2c0a3a9304e3b99a2c4d9bc9ebd52 P b8ecd9e070a5486f89f0a1e92c524318 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:08:45.191139 32177 tablet_replica.cc:330] T 03d2c0a3a9304e3b99a2c4d9bc9ebd52 P 387d7d4d48124399b1ad4dd2ab7d4589: stopping tablet replica
I20260808 04:08:45.191515 32177 raft_consensus.cc:2229] T 03d2c0a3a9304e3b99a2c4d9bc9ebd52 P 387d7d4d48124399b1ad4dd2ab7d4589 [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:08:45.192169 32177 raft_consensus.cc:2258] T 03d2c0a3a9304e3b99a2c4d9bc9ebd52 P 387d7d4d48124399b1ad4dd2ab7d4589 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:08:45.194182 32177 ts_tablet_manager.cc:1905] T 03d2c0a3a9304e3b99a2c4d9bc9ebd52 P 387d7d4d48124399b1ad4dd2ab7d4589: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:08:45.194481 32131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:45.174266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51786 (local address 127.5.83.1:33765)
0808 04:08:45.174435 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:45.174441 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:45.184242 (+ 9801us) server_negotiation.cc:407] Connection header received
0808 04:08:45.184656 (+ 414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:45.184662 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:45.184790 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:45.184863 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:08:45.185551 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:45.186816 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:45.189006 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:45.189702 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:45.189793 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:45.192816 (+ 3023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:08:45.192827 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:08:45.192842 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:08:45.193532 (+ 690us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:08:45.193587 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:08:45.193612 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:08:45.193704 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:45.193966 (+ 262us) server_negotiation.cc:299] Negotiation successful
0808 04:08:45.194076 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 04:08:45.195581 31716 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:08:45.201561 32177 ts_tablet_manager.cc:1918] T 03d2c0a3a9304e3b99a2c4d9bc9ebd52 P 387d7d4d48124399b1ad4dd2ab7d4589: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:08:45.201793 32177 log.cc:1196] T 03d2c0a3a9304e3b99a2c4d9bc9ebd52 P 387d7d4d48124399b1ad4dd2ab7d4589: Deleting WAL directory at /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161826605655-5452-0/minicluster-data/ts-2/wal/wals/03d2c0a3a9304e3b99a2c4d9bc9ebd52
I20260808 04:08:45.202553 32177 ts_tablet_manager.cc:1939] T 03d2c0a3a9304e3b99a2c4d9bc9ebd52 P 387d7d4d48124399b1ad4dd2ab7d4589: Deleting consensus metadata
I20260808 04:08:45.203408 32173 ts_tablet_manager.cc:1905] T 03d2c0a3a9304e3b99a2c4d9bc9ebd52 P b8ecd9e070a5486f89f0a1e92c524318: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:08:45.207859 31430 catalog_manager.cc:4841] TS 387d7d4d48124399b1ad4dd2ab7d4589 (127.5.83.3:39777): tablet 03d2c0a3a9304e3b99a2c4d9bc9ebd52 (table test-table [id=36d6cc5a5d334436b8f7857f7e1f58da]) successfully deleted
I20260808 04:08:45.209702 32173 ts_tablet_manager.cc:1918] T 03d2c0a3a9304e3b99a2c4d9bc9ebd52 P b8ecd9e070a5486f89f0a1e92c524318: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:08:45.209860 32173 log.cc:1196] T 03d2c0a3a9304e3b99a2c4d9bc9ebd52 P b8ecd9e070a5486f89f0a1e92c524318: Deleting WAL directory at /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161826605655-5452-0/minicluster-data/ts-1/wal/wals/03d2c0a3a9304e3b99a2c4d9bc9ebd52
I20260808 04:08:45.210371 32173 ts_tablet_manager.cc:1939] T 03d2c0a3a9304e3b99a2c4d9bc9ebd52 P b8ecd9e070a5486f89f0a1e92c524318: Deleting consensus metadata
I20260808 04:08:45.212105 31431 catalog_manager.cc:4841] TS b8ecd9e070a5486f89f0a1e92c524318 (127.5.83.2:33041): tablet 03d2c0a3a9304e3b99a2c4d9bc9ebd52 (table test-table [id=36d6cc5a5d334436b8f7857f7e1f58da]) successfully deleted
I20260808 04:08:45.293421 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 31538
I20260808 04:08:45.316628 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 31727
W20260808 04:08:45.316632 31432 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.83.1:33765 (error 108)
I20260808 04:08:45.318413 32127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:45.317969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:33765 (local address 127.0.0.1:51792)
0808 04:08:45.318114 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:45.318210 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.1:33765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260808 04:08:45.318750 31432 catalog_manager.cc:4584] TS c79d91a0da0e4ed1b8418b97a8e5802a (127.5.83.1:33765): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 03d2c0a3a9304e3b99a2c4d9bc9ebd52: Network error: Client connection negotiation failed: client connection to 127.5.83.1:33765: connect: Connection refused (error 111)
I20260808 04:08:45.324163 32135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:45.317726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40924 (local address 127.5.83.2:33041)
0808 04:08:45.317901 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:45.317905 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:45.317923 (+ 18us) server_negotiation.cc:407] Connection header received
0808 04:08:45.318393 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:45.318399 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:45.318537 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:45.318648 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:08:45.319093 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:45.319891 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:45.321143 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:45.321534 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:45.321644 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:45.322668 (+ 1024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:08:45.322677 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:08:45.322690 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:08:45.323247 (+ 557us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:08:45.323291 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:08:45.323311 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:08:45.323411 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:45.323542 (+ 131us) server_negotiation.cc:299] Negotiation successful
0808 04:08:45.323650 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 04:08:45.324738 31905 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:08:45.372339 32127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:45.371791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:33765 (local address 127.0.0.1:51808)
0808 04:08:45.372069 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:45.372169 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.1:33765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157}
I20260808 04:08:45.373847 32127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:45.373385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:33765 (local address 127.0.0.1:51816)
0808 04:08:45.373588 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:45.373671 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.1:33765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260808 04:08:45.425292 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 31727
I20260808 04:08:45.434738 32127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:45.434195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:33765 (local address 127.0.0.1:51832)
0808 04:08:45.434437 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:45.434542 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.1:33765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260808 04:08:45.436008 32127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:45.435533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:33765 (local address 127.0.0.1:51846)
0808 04:08:45.435738 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:45.435834 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.1:33765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260808 04:08:45.447085 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 31919
I20260808 04:08:45.454145 32137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:45.447847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56364 (local address 127.5.83.3:39777)
0808 04:08:45.448019 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:45.448025 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:45.448097 (+ 72us) server_negotiation.cc:407] Connection header received
0808 04:08:45.448302 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:45.448307 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:45.448396 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:45.448488 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:08:45.448947 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:45.449689 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:45.450936 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:45.451386 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:45.451496 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:45.452722 (+ 1226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:08:45.452731 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:08:45.452745 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:08:45.453391 (+ 646us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:08:45.453442 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:08:45.453462 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:08:45.453557 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:45.453715 (+ 158us) server_negotiation.cc:299] Negotiation successful
0808 04:08:45.453812 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 04:08:45.454895 32105 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:08:45.516633 32127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:45.516084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:33765 (local address 127.0.0.1:51860)
0808 04:08:45.516310 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:45.516425 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.1:33765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260808 04:08:45.517714 32127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:45.517261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:33765 (local address 127.0.0.1:51868)
0808 04:08:45.517489 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:45.517574 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.1:33765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260808 04:08:45.572535 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 31919
I20260808 04:08:45.596832 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 31410
I20260808 04:08:45.597922 31488 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:08:45.703219 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 31410
2026-08-08T04:08:45Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2860 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:08:45 dist-test-slave-1l3l krb5kdc[32190](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:08:45 dist-test-slave-1l3l krb5kdc[32190](info): set up 2 sockets
Aug 08 04:08:45 dist-test-slave-1l3l krb5kdc[32190](info): commencing operation
krb5kdc: starting...
W20260808 04:08:46.905619 5452 mini_kdc.cc:121] Time spent starting KDC: real 1.160s user 0.004s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/krb5kdc/test-user.keytab.
Aug 08 04:08:47 dist-test-slave-1l3l krb5kdc[32190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:08:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:08:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.83.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab.
Entry for principal kudu/127.5.83.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab.
I20260808 04:08:47.162660 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:37989
--webserver_interface=127.5.83.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab
--principal=kudu/127.5.83.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:38791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.62:37989
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:08:47.483803 32207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:08:47.484051 32207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:08:47.484176 32207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:08:47.491001 32207 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:08:47.491106 32207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:08:47.491183 32207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:08:47.491253 32207 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:08:47.491312 32207 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:08:47.500605 32207 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:38791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.62:37989
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab
--principal=kudu/127.5.83.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:37989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:08:47.501730 32207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:08:47.502940 32207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:08:47.513805 32214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:08:47.513851 32212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:08:47.513876 32219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:08:47.514562 32207 server_base.cc:1034] running on GCE node
I20260808 04:08:47.515281 32207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:08:47.516597 32207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:08:47.517791 32207 hybrid_clock.cc:648] HybridClock initialized: now 1786162127517749 us; error 26 us; skew 500 ppm
I20260808 04:08:47.518078 32207 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:08:47 dist-test-slave-1l3l krb5kdc[32190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162127, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:08:47.520956 32207 init.cc:377] Logged in from keytab as kudu/127.5.83.62@KRBTEST.COM (short username kudu)
I20260808 04:08:47.522291 32207 webserver.cc:460] Webserver started at http://127.5.83.62:45035/ using document root <none> and password file <none>
I20260808 04:08:47.522890 32207 fs_manager.cc:359] Metadata directory not provided
I20260808 04:08:47.522986 32207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:08:47.523224 32207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:08:47.525990 32207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/minicluster-data/master-0/data/instance:
uuid: "5160605195f149b1bdbd8ca5436050f9"
format_stamp: "Formatted at 2026-08-08 04:08:47 on dist-test-slave-1l3l"
server_key: "faf69963f5276bded18d63d0e89eef3d"
server_key_iv: "255b9b22b2b95c414cfa979a5c0f0267"
server_key_version: "encryptionkey@0"
I20260808 04:08:47.526666 32207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/minicluster-data/master-0/wal/instance:
uuid: "5160605195f149b1bdbd8ca5436050f9"
format_stamp: "Formatted at 2026-08-08 04:08:47 on dist-test-slave-1l3l"
server_key: "faf69963f5276bded18d63d0e89eef3d"
server_key_iv: "255b9b22b2b95c414cfa979a5c0f0267"
server_key_version: "encryptionkey@0"
I20260808 04:08:47.531284 32207 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260808 04:08:47.534436 32227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:08:47.535615 32207 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:08:47.535784 32207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/minicluster-data/master-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/minicluster-data/master-0/wal
uuid: "5160605195f149b1bdbd8ca5436050f9"
format_stamp: "Formatted at 2026-08-08 04:08:47 on dist-test-slave-1l3l"
server_key: "faf69963f5276bded18d63d0e89eef3d"
server_key_iv: "255b9b22b2b95c414cfa979a5c0f0267"
server_key_version: "encryptionkey@0"
I20260808 04:08:47.535948 32207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:08:47.548840 32207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:08:47.552282 32207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:08:47.552546 32207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:08:47.569284 32207 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.62:37989
I20260808 04:08:47.569305 32301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.62:37989 every 8 connection(s)
I20260808 04:08:47.570560 32207 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
I20260808 04:08:47.573999 32302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:08:47.578302 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 32207
I20260808 04:08:47.578504 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/minicluster-data/master-0/wal/instance
I20260808 04:08:47.579053 5452 external_mini_cluster.cc:1428] Setting key d0dcb349df0d41f4fba749fac2b4c517
I20260808 04:08:47.584739 32302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5160605195f149b1bdbd8ca5436050f9: Bootstrap starting.
I20260808 04:08:47.588088 32302 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5160605195f149b1bdbd8ca5436050f9: Neither blocks nor log segments found. Creating new log.
I20260808 04:08:47.589246 32302 log.cc:824] T 00000000000000000000000000000000 P 5160605195f149b1bdbd8ca5436050f9: Log is configured to *not* fsync() on all Append() calls
I20260808 04:08:47.592444 32302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5160605195f149b1bdbd8ca5436050f9: No bootstrap required, opened a new log
Aug 08 04:08:47 dist-test-slave-1l3l krb5kdc[32190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:08:47.600970 32302 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5160605195f149b1bdbd8ca5436050f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5160605195f149b1bdbd8ca5436050f9" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 37989 } }
I20260808 04:08:47.601464 32302 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5160605195f149b1bdbd8ca5436050f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:08:47.601553 32302 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5160605195f149b1bdbd8ca5436050f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5160605195f149b1bdbd8ca5436050f9, State: Initialized, Role: FOLLOWER
I20260808 04:08:47.602063 32302 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5160605195f149b1bdbd8ca5436050f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5160605195f149b1bdbd8ca5436050f9" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 37989 } }
I20260808 04:08:47.602272 32302 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5160605195f149b1bdbd8ca5436050f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:08:47.602401 32302 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5160605195f149b1bdbd8ca5436050f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:08:47.602512 32306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:47.580820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35818 (local address 127.5.83.62:37989)
0808 04:08:47.581635 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:47.581648 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:47.581692 (+ 44us) server_negotiation.cc:407] Connection header received
0808 04:08:47.583472 (+ 1780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:47.583494 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:47.583888 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:47.584200 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:08:47.586018 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:47.587332 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:47.589127 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:47.590026 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:47.590222 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:47.595106 (+ 4884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:08:47.595114 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:08:47.595127 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:08:47.595167 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:08:47.598237 (+ 3070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:08:47.599070 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:08:47.599075 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:08:47.599081 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:08:47.599260 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:08:47.599726 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:08:47.599733 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:08:47.599737 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:08:47.600307 (+ 570us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:08:47.600533 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:47.600972 (+ 439us) server_negotiation.cc:299] Negotiation successful
0808 04:08:47.601278 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":363,"threads_started":1}
I20260808 04:08:47.602578 32302 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5160605195f149b1bdbd8ca5436050f9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:08:47.604843 32302 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5160605195f149b1bdbd8ca5436050f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5160605195f149b1bdbd8ca5436050f9" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 37989 } }
I20260808 04:08:47.605294 32302 leader_election.cc:302] T 00000000000000000000000000000000 P 5160605195f149b1bdbd8ca5436050f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5160605195f149b1bdbd8ca5436050f9; no voters:
I20260808 04:08:47.605789 32302 leader_election.cc:288] T 00000000000000000000000000000000 P 5160605195f149b1bdbd8ca5436050f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:08:47.605916 32308 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5160605195f149b1bdbd8ca5436050f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:08:47.606647 32308 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5160605195f149b1bdbd8ca5436050f9 [term 1 LEADER]: Becoming Leader. State: Replica: 5160605195f149b1bdbd8ca5436050f9, State: Running, Role: LEADER
I20260808 04:08:47.607149 32308 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5160605195f149b1bdbd8ca5436050f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5160605195f149b1bdbd8ca5436050f9" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 37989 } }
I20260808 04:08:47.607784 32302 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5160605195f149b1bdbd8ca5436050f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:08:47.610107 32310 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5160605195f149b1bdbd8ca5436050f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5160605195f149b1bdbd8ca5436050f9. Latest consensus state: current_term: 1 leader_uuid: "5160605195f149b1bdbd8ca5436050f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5160605195f149b1bdbd8ca5436050f9" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 37989 } } }
I20260808 04:08:47.610236 32309 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5160605195f149b1bdbd8ca5436050f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5160605195f149b1bdbd8ca5436050f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5160605195f149b1bdbd8ca5436050f9" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 37989 } } }
I20260808 04:08:47.610566 32310 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5160605195f149b1bdbd8ca5436050f9 [sys.catalog]: This master's current role is: LEADER
I20260808 04:08:47.610574 32309 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5160605195f149b1bdbd8ca5436050f9 [sys.catalog]: This master's current role is: LEADER
I20260808 04:08:47.611423 32319 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:08:47.616938 32319 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:08:47.624858 32319 catalog_manager.cc:1347] Generated new cluster ID: ec4e94d0637846f89e19b37408dec1d3
I20260808 04:08:47.624962 32319 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:08:47.640106 32319 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:08:47.641482 32319 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:08:47.650930 32319 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5160605195f149b1bdbd8ca5436050f9: Generated new TSK 0
I20260808 04:08:47.652130 32319 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.83.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.1.keytab.
Entry for principal kudu/127.5.83.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.1.keytab.
I20260808 04:08:47.721992 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.1:0
--local_ip_for_outbound_sockets=127.5.83.1
--webserver_interface=127.5.83.1
--webserver_port=0
--tserver_master_addrs=127.5.83.62:37989
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.1.keytab
--principal=kudu/127.5.83.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:38791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:08:48.055539 32339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:08:48.055909 32339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:08:48.056031 32339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:08:48.063063 32339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:08:48.063159 32339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:08:48.063273 32339 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.1
I20260808 04:08:48.072517 32339 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:38791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.1.keytab
--principal=kudu/127.5.83.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.83.1
--webserver_port=0
--tserver_master_addrs=127.5.83.62:37989
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.1
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:08:48.073612 32339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:08:48.074815 32339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:08:48.087240 32346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:08:48.087280 32351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:08:48.087208 32348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:08:48.088040 32339 server_base.cc:1034] running on GCE node
I20260808 04:08:48.088629 32339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:08:48.089553 32339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:08:48.090754 32339 hybrid_clock.cc:648] HybridClock initialized: now 1786162128090708 us; error 35 us; skew 500 ppm
I20260808 04:08:48.091051 32339 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:08:48 dist-test-slave-1l3l krb5kdc[32190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162128, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:08:48.094065 32339 init.cc:377] Logged in from keytab as kudu/127.5.83.1@KRBTEST.COM (short username kudu)
I20260808 04:08:48.095482 32339 webserver.cc:460] Webserver started at http://127.5.83.1:44983/ using document root <none> and password file <none>
I20260808 04:08:48.096179 32339 fs_manager.cc:359] Metadata directory not provided
I20260808 04:08:48.096297 32339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:08:48.096593 32339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:08:48.099396 32339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/minicluster-data/ts-0/data/instance:
uuid: "c542981ff53d422ba9a9eb6e0322a440"
format_stamp: "Formatted at 2026-08-08 04:08:48 on dist-test-slave-1l3l"
server_key: "6feddac34cb24267fd4ba66e9416c101"
server_key_iv: "1f05e373e599fff603f92e001d513f50"
server_key_version: "encryptionkey@0"
I20260808 04:08:48.100188 32339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/minicluster-data/ts-0/wal/instance:
uuid: "c542981ff53d422ba9a9eb6e0322a440"
format_stamp: "Formatted at 2026-08-08 04:08:48 on dist-test-slave-1l3l"
server_key: "6feddac34cb24267fd4ba66e9416c101"
server_key_iv: "1f05e373e599fff603f92e001d513f50"
server_key_version: "encryptionkey@0"
I20260808 04:08:48.105199 32339 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 04:08:48.108465 32359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:08:48.109956 32339 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:08:48.110101 32339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/minicluster-data/ts-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/minicluster-data/ts-0/wal
uuid: "c542981ff53d422ba9a9eb6e0322a440"
format_stamp: "Formatted at 2026-08-08 04:08:48 on dist-test-slave-1l3l"
server_key: "6feddac34cb24267fd4ba66e9416c101"
server_key_iv: "1f05e373e599fff603f92e001d513f50"
server_key_version: "encryptionkey@0"
I20260808 04:08:48.110291 32339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:08:48.126864 32339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:08:48.130690 32339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:08:48.130923 32339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:08:48.131871 32339 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:08:48.133144 32339 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:08:48.133217 32339 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:08:48.133335 32339 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:08:48.133379 32339 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:08:48.164902 32339 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.1:38463
I20260808 04:08:48.164965 32530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.1:38463 every 8 connection(s)
I20260808 04:08:48.166113 32339 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
I20260808 04:08:48.176259 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 32339
I20260808 04:08:48.176481 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/minicluster-data/ts-0/wal/instance
I20260808 04:08:48.177011 5452 external_mini_cluster.cc:1428] Setting key 45c7f0e96698684dd7618c44be3ceb2b
Aug 08 04:08:48 dist-test-slave-1l3l krb5kdc[32190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162128, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.1@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:08:48.184638 32536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:48.168460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.1:40929 (local address 127.5.83.62:37989)
0808 04:08:48.168878 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:48.168882 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:48.169798 (+ 916us) server_negotiation.cc:407] Connection header received
0808 04:08:48.171507 (+ 1709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:48.171511 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:48.171577 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:48.171670 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:08:48.173073 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:48.173809 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:48.175235 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:48.175644 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:48.175930 (+ 286us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:48.179266 (+ 3336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:08:48.179271 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:08:48.179274 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:08:48.179304 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:08:48.181803 (+ 2499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:08:48.182481 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:08:48.182487 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:08:48.182490 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:08:48.182610 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:08:48.183003 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:08:48.183007 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:08:48.183011 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:08:48.183260 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:08:48.183390 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:48.184077 (+ 687us) server_negotiation.cc:299] Negotiation successful
0808 04:08:48.184292 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":210,"threads_started":1}
I20260808 04:08:48.185495 32537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:48.168958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:37989 (local address 127.5.83.1:40929)
0808 04:08:48.169641 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:48.169686 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 04:08:48.171272 (+ 1586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:08:48.171881 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:08:48.171889 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:08:48.172411 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:08:48.172929 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:48.172946 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:48.173944 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:48.173947 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:48.175127 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:48.175139 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:48.175971 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:48.175976 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:48.176163 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:48.176890 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:08:48.176913 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:08:48.179066 (+ 2153us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:08:48.181937 (+ 2871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:08:48.181946 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:08:48.181960 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:08:48.182360 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:08:48.182714 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:08:48.182719 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:08:48.182722 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:08:48.182896 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:08:48.183384 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:08:48.183391 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:08:48.183683 (+ 292us) client_negotiation.cc:769] Sending connection context
0808 04:08:48.183968 (+ 285us) client_negotiation.cc:240] Negotiation successful
0808 04:08:48.184439 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":261,"threads_started":1}
I20260808 04:08:48.186877 32531 heartbeater.cc:347] Connected to a master server at 127.5.83.62:37989
I20260808 04:08:48.187145 32531 heartbeater.cc:464] Registering TS with master...
I20260808 04:08:48.187851 32531 heartbeater.cc:511] Master 127.5.83.62:37989 requested a full tablet report, sending...
I20260808 04:08:48.189949 32256 ts_manager.cc:194] Registered new tserver with Master: c542981ff53d422ba9a9eb6e0322a440 (127.5.83.1:38463)
I20260808 04:08:48.191392 32256 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.1@KRBTEST.COM'} at 127.5.83.1:40929
WARNING: no policy specified for kudu/127.5.83.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.2.keytab.
Entry for principal kudu/127.5.83.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.2.keytab.
I20260808 04:08:48.234647 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.2:0
--local_ip_for_outbound_sockets=127.5.83.2
--webserver_interface=127.5.83.2
--webserver_port=0
--tserver_master_addrs=127.5.83.62:37989
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.2.keytab
--principal=kudu/127.5.83.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:38791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:08:48.557866 32540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:08:48.558144 32540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:08:48.558279 32540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:08:48.565219 32540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:08:48.565326 32540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:08:48.565472 32540 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.2
I20260808 04:08:48.574648 32540 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:38791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.2.keytab
--principal=kudu/127.5.83.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.83.2
--webserver_port=0
--tserver_master_addrs=127.5.83.62:37989
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.2
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:08:48.575816 32540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:08:48.577036 32540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:08:48.589620 32546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:08:48.590329 32547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:08:48.590801 32549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:08:48.591202 32540 server_base.cc:1034] running on GCE node
I20260808 04:08:48.591804 32540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:08:48.592677 32540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:08:48.593866 32540 hybrid_clock.cc:648] HybridClock initialized: now 1786162128593810 us; error 50 us; skew 500 ppm
I20260808 04:08:48.594188 32540 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:08:48 dist-test-slave-1l3l krb5kdc[32190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162128, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:08:48.602095 32540 init.cc:377] Logged in from keytab as kudu/127.5.83.2@KRBTEST.COM (short username kudu)
I20260808 04:08:48.603421 32540 webserver.cc:460] Webserver started at http://127.5.83.2:36923/ using document root <none> and password file <none>
I20260808 04:08:48.604092 32540 fs_manager.cc:359] Metadata directory not provided
I20260808 04:08:48.604205 32540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:08:48.604498 32540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:08:48.607302 32540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/minicluster-data/ts-1/data/instance:
uuid: "3640a584dd9d412facc2e8d05c476820"
format_stamp: "Formatted at 2026-08-08 04:08:48 on dist-test-slave-1l3l"
server_key: "708fadb06b488447334f09ed8608ef94"
server_key_iv: "d9cb98019c9d5e6f7b550e3b454574d8"
server_key_version: "encryptionkey@0"
I20260808 04:08:48.608084 32540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/minicluster-data/ts-1/wal/instance:
uuid: "3640a584dd9d412facc2e8d05c476820"
format_stamp: "Formatted at 2026-08-08 04:08:48 on dist-test-slave-1l3l"
server_key: "708fadb06b488447334f09ed8608ef94"
server_key_iv: "d9cb98019c9d5e6f7b550e3b454574d8"
server_key_version: "encryptionkey@0"
I20260808 04:08:48.613082 32540 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260808 04:08:48.616490 32559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:08:48.617671 32540 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:08:48.617827 32540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/minicluster-data/ts-1/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/minicluster-data/ts-1/wal
uuid: "3640a584dd9d412facc2e8d05c476820"
format_stamp: "Formatted at 2026-08-08 04:08:48 on dist-test-slave-1l3l"
server_key: "708fadb06b488447334f09ed8608ef94"
server_key_iv: "d9cb98019c9d5e6f7b550e3b454574d8"
server_key_version: "encryptionkey@0"
I20260808 04:08:48.618021 32540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:08:48.640609 32540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:08:48.644392 32540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:08:48.644655 32540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:08:48.645588 32540 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:08:48.646899 32540 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:08:48.646992 32540 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:08:48.647090 32540 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:08:48.647133 32540 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:08:48.676738 32540 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.2:35763
I20260808 04:08:48.676801 32729 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.2:35763 every 8 connection(s)
I20260808 04:08:48.677935 32540 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
I20260808 04:08:48.687867 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 32540
I20260808 04:08:48.688064 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/minicluster-data/ts-1/wal/instance
I20260808 04:08:48.688508 5452 external_mini_cluster.cc:1428] Setting key 5aa5879a4162ae6d196523c7ac22c5be
Aug 08 04:08:48 dist-test-slave-1l3l krb5kdc[32190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162128, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.2@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:08:48.696674 32536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:48.680292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.2:33897 (local address 127.5.83.62:37989)
0808 04:08:48.680459 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:48.680464 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:48.681530 (+ 1066us) server_negotiation.cc:407] Connection header received
0808 04:08:48.683286 (+ 1756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:48.683291 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:48.683382 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:48.683471 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:08:48.684954 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:48.685652 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:48.687271 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:48.687868 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:48.688012 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:48.691460 (+ 3448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:08:48.691467 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:08:48.691470 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:08:48.691502 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:08:48.693868 (+ 2366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:08:48.694526 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:08:48.694530 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:08:48.694532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:08:48.694653 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:08:48.695044 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:08:48.695049 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:08:48.695052 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:08:48.695293 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:08:48.695441 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:48.696099 (+ 658us) server_negotiation.cc:299] Negotiation successful
0808 04:08:48.696349 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 04:08:48.697525 32733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:48.680675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:37989 (local address 127.5.83.2:33897)
0808 04:08:48.681354 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:48.681407 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0808 04:08:48.683026 (+ 1619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:08:48.683720 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:08:48.683733 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:08:48.684244 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:08:48.684776 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:48.684794 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:48.685803 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:48.685807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:48.687128 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:48.687140 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:48.688117 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:48.688125 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:48.688334 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:48.688960 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:08:48.688981 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:08:48.691256 (+ 2275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:08:48.694002 (+ 2746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:08:48.694010 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:08:48.694024 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:08:48.694417 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:08:48.694754 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:08:48.694759 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:08:48.694763 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:08:48.694940 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:08:48.695442 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:08:48.695450 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:08:48.695758 (+ 308us) client_negotiation.cc:769] Sending connection context
0808 04:08:48.695995 (+ 237us) client_negotiation.cc:240] Negotiation successful
0808 04:08:48.696468 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":277,"threads_started":1}
I20260808 04:08:48.698807 32731 heartbeater.cc:347] Connected to a master server at 127.5.83.62:37989
I20260808 04:08:48.699144 32731 heartbeater.cc:464] Registering TS with master...
I20260808 04:08:48.699846 32731 heartbeater.cc:511] Master 127.5.83.62:37989 requested a full tablet report, sending...
I20260808 04:08:48.701203 32256 ts_manager.cc:194] Registered new tserver with Master: 3640a584dd9d412facc2e8d05c476820 (127.5.83.2:35763)
I20260808 04:08:48.702126 32256 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.2@KRBTEST.COM'} at 127.5.83.2:33897
WARNING: no policy specified for kudu/127.5.83.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.3.keytab.
Entry for principal kudu/127.5.83.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.3.keytab.
I20260808 04:08:48.746987 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.3:0
--local_ip_for_outbound_sockets=127.5.83.3
--webserver_interface=127.5.83.3
--webserver_port=0
--tserver_master_addrs=127.5.83.62:37989
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.3.keytab
--principal=kudu/127.5.83.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:38791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:08:49.069571 32736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:08:49.069828 32736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:08:49.069962 32736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:08:49.076956 32736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:08:49.077059 32736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:08:49.077211 32736 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.3
I20260808 04:08:49.086639 32736 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:38791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.3.keytab
--principal=kudu/127.5.83.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.83.3
--webserver_port=0
--tserver_master_addrs=127.5.83.62:37989
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.3
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:08:49.087891 32736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:08:49.089133 32736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:08:49.101485 32745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:08:49.101545 32746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:08:49.101565 32749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:08:49.102062 32736 server_base.cc:1034] running on GCE node
I20260808 04:08:49.102620 32736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:08:49.103495 32736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:08:49.104691 32736 hybrid_clock.cc:648] HybridClock initialized: now 1786162129104622 us; error 61 us; skew 500 ppm
I20260808 04:08:49.104983 32736 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:08:49 dist-test-slave-1l3l krb5kdc[32190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162129, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:08:49.108013 32736 init.cc:377] Logged in from keytab as kudu/127.5.83.3@KRBTEST.COM (short username kudu)
I20260808 04:08:49.109385 32736 webserver.cc:460] Webserver started at http://127.5.83.3:41955/ using document root <none> and password file <none>
I20260808 04:08:49.110010 32736 fs_manager.cc:359] Metadata directory not provided
I20260808 04:08:49.110121 32736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:08:49.110404 32736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:08:49.113530 32736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/minicluster-data/ts-2/data/instance:
uuid: "c6501645d9f54ec4b222ac740e56c479"
format_stamp: "Formatted at 2026-08-08 04:08:49 on dist-test-slave-1l3l"
server_key: "94e1f6c69b9dd7d244f59c0065a549c5"
server_key_iv: "6fd5a8e755d36fe3c35af8213d287cf1"
server_key_version: "encryptionkey@0"
I20260808 04:08:49.114290 32736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/minicluster-data/ts-2/wal/instance:
uuid: "c6501645d9f54ec4b222ac740e56c479"
format_stamp: "Formatted at 2026-08-08 04:08:49 on dist-test-slave-1l3l"
server_key: "94e1f6c69b9dd7d244f59c0065a549c5"
server_key_iv: "6fd5a8e755d36fe3c35af8213d287cf1"
server_key_version: "encryptionkey@0"
I20260808 04:08:49.119158 32736 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260808 04:08:49.122485 32762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:08:49.123674 32736 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:08:49.123852 32736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/minicluster-data/ts-2/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/minicluster-data/ts-2/wal
uuid: "c6501645d9f54ec4b222ac740e56c479"
format_stamp: "Formatted at 2026-08-08 04:08:49 on dist-test-slave-1l3l"
server_key: "94e1f6c69b9dd7d244f59c0065a549c5"
server_key_iv: "6fd5a8e755d36fe3c35af8213d287cf1"
server_key_version: "encryptionkey@0"
I20260808 04:08:49.124048 32736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:08:49.144357 32736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:08:49.148139 32736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:08:49.148427 32736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:08:49.149407 32736 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:08:49.150696 32736 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:08:49.150774 32736 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:08:49.150897 32736 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:08:49.150947 32736 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:08:49.181058 32736 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.3:33673
I20260808 04:08:49.181113 461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.3:33673 every 8 connection(s)
I20260808 04:08:49.182214 32736 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
I20260808 04:08:49.189849 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 32736
I20260808 04:08:49.190011 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/minicluster-data/ts-2/wal/instance
I20260808 04:08:49.190446 5452 external_mini_cluster.cc:1428] Setting key becbdcecb1b7fdf86edfb62a4f8f63ef
I20260808 04:08:49.193936 32531 heartbeater.cc:503] Master 127.5.83.62:37989 was elected leader, sending a full tablet report...
Aug 08 04:08:49 dist-test-slave-1l3l krb5kdc[32190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162129, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.3@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:08:49.201340 32536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:49.184517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.3:55573 (local address 127.5.83.62:37989)
0808 04:08:49.184722 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:49.184728 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:49.185793 (+ 1065us) server_negotiation.cc:407] Connection header received
0808 04:08:49.187577 (+ 1784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:49.187583 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:49.187662 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:49.187797 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:08:49.189222 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:49.190066 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:49.191904 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:49.192322 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:49.192437 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:49.195941 (+ 3504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:08:49.195946 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:08:49.195949 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:08:49.195981 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:08:49.198269 (+ 2288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:08:49.198912 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:08:49.198917 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:08:49.198921 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:08:49.199049 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:08:49.199438 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:08:49.199442 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:08:49.199444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:08:49.199699 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:08:49.199867 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:49.200653 (+ 786us) server_negotiation.cc:299] Negotiation successful
0808 04:08:49.201011 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 04:08:49.202227 465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:49.184956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:37989 (local address 127.5.83.3:55573)
0808 04:08:49.185634 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:49.185674 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 04:08:49.187319 (+ 1645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:08:49.188011 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:08:49.188019 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:08:49.188516 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:08:49.189053 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:49.189071 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:49.190210 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:49.190215 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:49.191767 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:49.191781 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:49.192539 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:49.192548 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:49.192776 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:49.193374 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:08:49.193400 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:08:49.195751 (+ 2351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:08:49.198394 (+ 2643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:08:49.198402 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:08:49.198415 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:08:49.198802 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:08:49.199151 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:08:49.199156 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:08:49.199160 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:08:49.199334 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:08:49.199859 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:08:49.199866 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:08:49.200171 (+ 305us) client_negotiation.cc:769] Sending connection context
0808 04:08:49.200530 (+ 359us) client_negotiation.cc:240] Negotiation successful
0808 04:08:49.201109 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":264,"threads_started":1}
I20260808 04:08:49.203393 462 heartbeater.cc:347] Connected to a master server at 127.5.83.62:37989
I20260808 04:08:49.203675 462 heartbeater.cc:464] Registering TS with master...
I20260808 04:08:49.204519 462 heartbeater.cc:511] Master 127.5.83.62:37989 requested a full tablet report, sending...
I20260808 04:08:49.205904 32256 ts_manager.cc:194] Registered new tserver with Master: c6501645d9f54ec4b222ac740e56c479 (127.5.83.3:33673)
I20260808 04:08:49.206664 32256 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.3@KRBTEST.COM'} at 127.5.83.3:55573
I20260808 04:08:49.217217 5452 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:08:49.234064 32536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:49.221255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53618 (local address 127.5.83.62:37989)
0808 04:08:49.221411 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:49.221415 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:49.221869 (+ 454us) server_negotiation.cc:407] Connection header received
0808 04:08:49.222034 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:49.222039 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:49.222129 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:49.222220 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:08:49.223643 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:49.224397 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:49.225721 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:49.226133 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:49.226241 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:49.229036 (+ 2795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:08:49.229043 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:08:49.229048 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:08:49.229081 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:08:49.231335 (+ 2254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:08:49.232131 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:08:49.232136 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:08:49.232140 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:08:49.232261 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:08:49.232691 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:08:49.232695 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:08:49.232697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:08:49.232995 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:08:49.233244 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:49.233640 (+ 396us) server_negotiation.cc:299] Negotiation successful
0808 04:08:49.233802 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 04:08:49.237727 32256 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53618:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:08:49.244513 32256 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:08:49.266937 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:49.256607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:38463 (local address 127.0.0.1:41708)
0808 04:08:49.258074 (+ 1467us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:49.258100 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0808 04:08:49.258210 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:08:49.259116 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:08:49.259121 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:08:49.259150 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:08:49.259735 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:49.259755 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:49.261555 (+ 1800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:49.261565 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:49.264587 (+ 3022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:49.264600 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:49.265945 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:49.265952 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:49.266132 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:49.266187 (+ 55us) client_negotiation.cc:769] Sending connection context
0808 04:08:49.266292 (+ 105us) client_negotiation.cc:240] Negotiation successful
0808 04:08:49.266474 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1347,"mutex_wait_us":154,"thread_start_us":1402,"threads_started":1}
I20260808 04:08:49.267331 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:49.256039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38336 (local address 127.5.83.2:35763)
0808 04:08:49.257289 (+ 1250us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:49.257299 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:49.257974 (+ 675us) server_negotiation.cc:407] Connection header received
0808 04:08:49.258317 (+ 343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:49.258325 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:49.258595 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:49.258768 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:08:49.259906 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:49.261439 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:49.264752 (+ 3313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:49.266038 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:49.266163 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:49.266340 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:49.266883 (+ 543us) server_negotiation.cc:299] Negotiation successful
0808 04:08:49.267045 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1090,"thread_start_us":992,"threads_started":1}
I20260808 04:08:49.267527 485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:49.258188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.3:33673 (local address 127.0.0.1:60998)
0808 04:08:49.258824 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:49.258860 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0808 04:08:49.260188 (+ 1328us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:08:49.261266 (+ 1078us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:08:49.261271 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:08:49.261298 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:08:49.261702 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:49.261715 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:49.263242 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:49.263248 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:49.264947 (+ 1699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:49.264962 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:49.266690 (+ 1728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:49.266697 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:49.266856 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:49.266889 (+ 33us) client_negotiation.cc:769] Sending connection context
0808 04:08:49.266950 (+ 61us) client_negotiation.cc:240] Negotiation successful
0808 04:08:49.267060 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"mutex_wait_us":3,"thread_start_us":796,"threads_started":1}
I20260808 04:08:49.267936 484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:49.256780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41708 (local address 127.5.83.1:38463)
0808 04:08:49.257816 (+ 1036us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:49.257824 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:49.258210 (+ 386us) server_negotiation.cc:407] Connection header received
0808 04:08:49.258431 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:49.258439 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:49.258664 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:49.258841 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:08:49.260018 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:49.261407 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:49.264727 (+ 3320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:49.265762 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:49.265889 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:49.266124 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:49.266356 (+ 232us) server_negotiation.cc:299] Negotiation successful
0808 04:08:49.266511 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":877,"thread_start_us":666,"threads_started":1}
I20260808 04:08:49.268033 481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:49.255839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.2:35763 (local address 127.0.0.1:38336)
0808 04:08:49.257787 (+ 1948us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:49.257846 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0808 04:08:49.258049 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:08:49.259064 (+ 1015us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:08:49.259071 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:08:49.259102 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:08:49.259668 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:49.259685 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:49.261583 (+ 1898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:49.261588 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:49.264554 (+ 2966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:49.264574 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:49.266495 (+ 1921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:49.266502 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:49.266672 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:49.266705 (+ 33us) client_negotiation.cc:769] Sending connection context
0808 04:08:49.267536 (+ 831us) client_negotiation.cc:240] Negotiation successful
0808 04:08:49.267657 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1677,"mutex_wait_us":358,"thread_start_us":1551,"threads_started":1}
I20260808 04:08:49.268646 486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:49.258700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60998 (local address 127.5.83.3:33673)
0808 04:08:49.260755 (+ 2055us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:49.260762 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:49.260793 (+ 31us) server_negotiation.cc:407] Connection header received
0808 04:08:49.260893 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:49.260900 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:49.261089 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:49.261256 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:08:49.261887 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:49.263071 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:49.265115 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:49.266473 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:49.267861 (+ 1388us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:49.268077 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:49.268147 (+ 70us) server_negotiation.cc:299] Negotiation successful
0808 04:08:49.268297 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1896,"thread_start_us":1777,"threads_started":1}
I20260808 04:08:49.270869 32636 tablet_service.cc:1467] Processing CreateTablet for tablet f496fabfd3f14a23b61368817358c88b (DEFAULT_TABLE table=test-table [id=938b45f2307548d5b6bed308bedb7c46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:08:49.270874 32428 tablet_service.cc:1467] Processing CreateTablet for tablet f496fabfd3f14a23b61368817358c88b (DEFAULT_TABLE table=test-table [id=938b45f2307548d5b6bed308bedb7c46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:08:49.272086 32636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f496fabfd3f14a23b61368817358c88b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:08:49.272092 32428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f496fabfd3f14a23b61368817358c88b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:08:49.274436 361 tablet_service.cc:1467] Processing CreateTablet for tablet f496fabfd3f14a23b61368817358c88b (DEFAULT_TABLE table=test-table [id=938b45f2307548d5b6bed308bedb7c46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:08:49.275449 361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f496fabfd3f14a23b61368817358c88b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:08:49.286623 488 tablet_bootstrap.cc:492] T f496fabfd3f14a23b61368817358c88b P 3640a584dd9d412facc2e8d05c476820: Bootstrap starting.
I20260808 04:08:49.288385 489 tablet_bootstrap.cc:492] T f496fabfd3f14a23b61368817358c88b P c542981ff53d422ba9a9eb6e0322a440: Bootstrap starting.
I20260808 04:08:49.289795 487 tablet_bootstrap.cc:492] T f496fabfd3f14a23b61368817358c88b P c6501645d9f54ec4b222ac740e56c479: Bootstrap starting.
I20260808 04:08:49.290396 488 tablet_bootstrap.cc:654] T f496fabfd3f14a23b61368817358c88b P 3640a584dd9d412facc2e8d05c476820: Neither blocks nor log segments found. Creating new log.
I20260808 04:08:49.291445 488 log.cc:824] T f496fabfd3f14a23b61368817358c88b P 3640a584dd9d412facc2e8d05c476820: Log is configured to *not* fsync() on all Append() calls
I20260808 04:08:49.291834 489 tablet_bootstrap.cc:654] T f496fabfd3f14a23b61368817358c88b P c542981ff53d422ba9a9eb6e0322a440: Neither blocks nor log segments found. Creating new log.
I20260808 04:08:49.293097 487 tablet_bootstrap.cc:654] T f496fabfd3f14a23b61368817358c88b P c6501645d9f54ec4b222ac740e56c479: Neither blocks nor log segments found. Creating new log.
I20260808 04:08:49.293148 489 log.cc:824] T f496fabfd3f14a23b61368817358c88b P c542981ff53d422ba9a9eb6e0322a440: Log is configured to *not* fsync() on all Append() calls
I20260808 04:08:49.294317 487 log.cc:824] T f496fabfd3f14a23b61368817358c88b P c6501645d9f54ec4b222ac740e56c479: Log is configured to *not* fsync() on all Append() calls
I20260808 04:08:49.294538 488 tablet_bootstrap.cc:492] T f496fabfd3f14a23b61368817358c88b P 3640a584dd9d412facc2e8d05c476820: No bootstrap required, opened a new log
I20260808 04:08:49.294793 488 ts_tablet_manager.cc:1397] T f496fabfd3f14a23b61368817358c88b P 3640a584dd9d412facc2e8d05c476820: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260808 04:08:49.297276 489 tablet_bootstrap.cc:492] T f496fabfd3f14a23b61368817358c88b P c542981ff53d422ba9a9eb6e0322a440: No bootstrap required, opened a new log
I20260808 04:08:49.297503 489 ts_tablet_manager.cc:1397] T f496fabfd3f14a23b61368817358c88b P c542981ff53d422ba9a9eb6e0322a440: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260808 04:08:49.297509 487 tablet_bootstrap.cc:492] T f496fabfd3f14a23b61368817358c88b P c6501645d9f54ec4b222ac740e56c479: No bootstrap required, opened a new log
I20260808 04:08:49.297737 487 ts_tablet_manager.cc:1397] T f496fabfd3f14a23b61368817358c88b P c6501645d9f54ec4b222ac740e56c479: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260808 04:08:49.304349 489 raft_consensus.cc:348] T f496fabfd3f14a23b61368817358c88b P c542981ff53d422ba9a9eb6e0322a440 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3640a584dd9d412facc2e8d05c476820" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 35763 } } peers { permanent_uuid: "c6501645d9f54ec4b222ac740e56c479" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 33673 } } peers { permanent_uuid: "c542981ff53d422ba9a9eb6e0322a440" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 38463 } }
I20260808 04:08:49.304816 489 raft_consensus.cc:374] T f496fabfd3f14a23b61368817358c88b P c542981ff53d422ba9a9eb6e0322a440 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:08:49.304956 489 raft_consensus.cc:729] T f496fabfd3f14a23b61368817358c88b P c542981ff53d422ba9a9eb6e0322a440 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c542981ff53d422ba9a9eb6e0322a440, State: Initialized, Role: FOLLOWER
I20260808 04:08:49.304999 488 raft_consensus.cc:348] T f496fabfd3f14a23b61368817358c88b P 3640a584dd9d412facc2e8d05c476820 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3640a584dd9d412facc2e8d05c476820" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 35763 } } peers { permanent_uuid: "c6501645d9f54ec4b222ac740e56c479" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 33673 } } peers { permanent_uuid: "c542981ff53d422ba9a9eb6e0322a440" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 38463 } }
I20260808 04:08:49.305483 488 raft_consensus.cc:374] T f496fabfd3f14a23b61368817358c88b P 3640a584dd9d412facc2e8d05c476820 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:08:49.305586 488 raft_consensus.cc:729] T f496fabfd3f14a23b61368817358c88b P 3640a584dd9d412facc2e8d05c476820 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3640a584dd9d412facc2e8d05c476820, State: Initialized, Role: FOLLOWER
I20260808 04:08:49.305675 489 consensus_queue.cc:260] T f496fabfd3f14a23b61368817358c88b P c542981ff53d422ba9a9eb6e0322a440 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3640a584dd9d412facc2e8d05c476820" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 35763 } } peers { permanent_uuid: "c6501645d9f54ec4b222ac740e56c479" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 33673 } } peers { permanent_uuid: "c542981ff53d422ba9a9eb6e0322a440" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 38463 } }
I20260808 04:08:49.306141 488 consensus_queue.cc:260] T f496fabfd3f14a23b61368817358c88b P 3640a584dd9d412facc2e8d05c476820 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3640a584dd9d412facc2e8d05c476820" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 35763 } } peers { permanent_uuid: "c6501645d9f54ec4b222ac740e56c479" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 33673 } } peers { permanent_uuid: "c542981ff53d422ba9a9eb6e0322a440" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 38463 } }
I20260808 04:08:49.307431 489 ts_tablet_manager.cc:1428] T f496fabfd3f14a23b61368817358c88b P c542981ff53d422ba9a9eb6e0322a440: Time spent starting tablet: real 0.010s user 0.004s sys 0.006s
I20260808 04:08:49.308470 32731 heartbeater.cc:503] Master 127.5.83.62:37989 was elected leader, sending a full tablet report...
I20260808 04:08:49.308343 487 raft_consensus.cc:348] T f496fabfd3f14a23b61368817358c88b P c6501645d9f54ec4b222ac740e56c479 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3640a584dd9d412facc2e8d05c476820" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 35763 } } peers { permanent_uuid: "c6501645d9f54ec4b222ac740e56c479" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 33673 } } peers { permanent_uuid: "c542981ff53d422ba9a9eb6e0322a440" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 38463 } }
I20260808 04:08:49.308796 487 raft_consensus.cc:374] T f496fabfd3f14a23b61368817358c88b P c6501645d9f54ec4b222ac740e56c479 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:08:49.308918 487 raft_consensus.cc:729] T f496fabfd3f14a23b61368817358c88b P c6501645d9f54ec4b222ac740e56c479 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6501645d9f54ec4b222ac740e56c479, State: Initialized, Role: FOLLOWER
I20260808 04:08:49.309111 488 ts_tablet_manager.cc:1428] T f496fabfd3f14a23b61368817358c88b P 3640a584dd9d412facc2e8d05c476820: Time spent starting tablet: real 0.014s user 0.012s sys 0.001s
I20260808 04:08:49.309669 487 consensus_queue.cc:260] T f496fabfd3f14a23b61368817358c88b P c6501645d9f54ec4b222ac740e56c479 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3640a584dd9d412facc2e8d05c476820" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 35763 } } peers { permanent_uuid: "c6501645d9f54ec4b222ac740e56c479" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 33673 } } peers { permanent_uuid: "c542981ff53d422ba9a9eb6e0322a440" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 38463 } }
I20260808 04:08:49.310735 462 heartbeater.cc:503] Master 127.5.83.62:37989 was elected leader, sending a full tablet report...
I20260808 04:08:49.311350 487 ts_tablet_manager.cc:1428] T f496fabfd3f14a23b61368817358c88b P c6501645d9f54ec4b222ac740e56c479: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
W20260808 04:08:49.418309 32532 tablet.cc:2369] T f496fabfd3f14a23b61368817358c88b P c542981ff53d422ba9a9eb6e0322a440: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 04:08:49.429715 32732 tablet.cc:2369] T f496fabfd3f14a23b61368817358c88b P 3640a584dd9d412facc2e8d05c476820: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 04:08:49.433694 463 tablet.cc:2369] T f496fabfd3f14a23b61368817358c88b P c6501645d9f54ec4b222ac740e56c479: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:08:49.497292 493 raft_consensus.cc:482] T f496fabfd3f14a23b61368817358c88b P c542981ff53d422ba9a9eb6e0322a440 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:08:49.497591 493 raft_consensus.cc:504] T f496fabfd3f14a23b61368817358c88b P c542981ff53d422ba9a9eb6e0322a440 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3640a584dd9d412facc2e8d05c476820" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 35763 } } peers { permanent_uuid: "c6501645d9f54ec4b222ac740e56c479" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 33673 } } peers { permanent_uuid: "c542981ff53d422ba9a9eb6e0322a440" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 38463 } }
I20260808 04:08:49.499316 493 leader_election.cc:288] T f496fabfd3f14a23b61368817358c88b P c542981ff53d422ba9a9eb6e0322a440 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3640a584dd9d412facc2e8d05c476820 (127.5.83.2:35763), c6501645d9f54ec4b222ac740e56c479 (127.5.83.3:33673)
I20260808 04:08:49.506384 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:49.499690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.1:38729 (local address 127.5.83.2:35763)
0808 04:08:49.499918 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:49.499922 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:49.500443 (+ 521us) server_negotiation.cc:407] Connection header received
0808 04:08:49.500672 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:49.500678 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:49.500758 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:49.500859 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:08:49.501598 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:49.502565 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:49.504506 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:49.505481 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:49.505582 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:49.505660 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:49.506025 (+ 365us) server_negotiation.cc:299] Negotiation successful
0808 04:08:49.506111 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260808 04:08:49.506448 497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:49.499668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.2:35763 (local address 127.5.83.1:38729)
0808 04:08:49.500354 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:49.500372 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 04:08:49.500514 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:08:49.501020 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:08:49.501025 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:08:49.501068 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:08:49.501442 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:49.501457 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:49.502688 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:49.502693 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:49.504360 (+ 1667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:49.504372 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:49.505688 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:49.505694 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:49.505864 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:49.505911 (+ 47us) client_negotiation.cc:769] Sending connection context
0808 04:08:49.506012 (+ 101us) client_negotiation.cc:240] Negotiation successful
0808 04:08:49.506134 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":373,"threads_started":1}
I20260808 04:08:49.506958 486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:49.500674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.1:53427 (local address 127.5.83.3:33673)
0808 04:08:49.500920 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:49.500927 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:49.501341 (+ 414us) server_negotiation.cc:407] Connection header received
0808 04:08:49.501504 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:49.501509 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:49.501594 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:49.501662 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:08:49.502257 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:49.503171 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:49.505118 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:49.506051 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:49.506156 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:49.506246 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:49.506621 (+ 375us) server_negotiation.cc:299] Negotiation successful
0808 04:08:49.506719 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260808 04:08:49.507452 499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:49.500521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.3:33673 (local address 127.5.83.1:53427)
0808 04:08:49.501239 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:49.501253 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:08:49.501361 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:08:49.501701 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:08:49.501706 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:08:49.501731 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:08:49.502100 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:49.502113 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:49.503302 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:49.503305 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:49.504967 (+ 1662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:49.504981 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:49.506269 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:49.506276 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:49.506427 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:49.506464 (+ 37us) client_negotiation.cc:769] Sending connection context
0808 04:08:49.507113 (+ 649us) client_negotiation.cc:240] Negotiation successful
0808 04:08:49.507225 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":373,"threads_started":1}
I20260808 04:08:49.507447 32668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f496fabfd3f14a23b61368817358c88b" candidate_uuid: "c542981ff53d422ba9a9eb6e0322a440" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3640a584dd9d412facc2e8d05c476820" is_pre_election: true
I20260808 04:08:49.508001 32668 raft_consensus.cc:2454] T f496fabfd3f14a23b61368817358c88b P 3640a584dd9d412facc2e8d05c476820 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c542981ff53d422ba9a9eb6e0322a440 in term 0.
I20260808 04:08:49.508378 386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f496fabfd3f14a23b61368817358c88b" candidate_uuid: "c542981ff53d422ba9a9eb6e0322a440" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6501645d9f54ec4b222ac740e56c479" is_pre_election: true
I20260808 04:08:49.508671 32360 leader_election.cc:302] T f496fabfd3f14a23b61368817358c88b P c542981ff53d422ba9a9eb6e0322a440 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3640a584dd9d412facc2e8d05c476820, c542981ff53d422ba9a9eb6e0322a440; no voters:
I20260808 04:08:49.508829 386 raft_consensus.cc:2454] T f496fabfd3f14a23b61368817358c88b P c6501645d9f54ec4b222ac740e56c479 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c542981ff53d422ba9a9eb6e0322a440 in term 0.
I20260808 04:08:49.509073 493 raft_consensus.cc:2781] T f496fabfd3f14a23b61368817358c88b P c542981ff53d422ba9a9eb6e0322a440 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:08:49.509207 493 raft_consensus.cc:482] T f496fabfd3f14a23b61368817358c88b P c542981ff53d422ba9a9eb6e0322a440 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:08:49.509323 493 raft_consensus.cc:3037] T f496fabfd3f14a23b61368817358c88b P c542981ff53d422ba9a9eb6e0322a440 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:08:49.511588 493 raft_consensus.cc:504] T f496fabfd3f14a23b61368817358c88b P c542981ff53d422ba9a9eb6e0322a440 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3640a584dd9d412facc2e8d05c476820" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 35763 } } peers { permanent_uuid: "c6501645d9f54ec4b222ac740e56c479" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 33673 } } peers { permanent_uuid: "c542981ff53d422ba9a9eb6e0322a440" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 38463 } }
I20260808 04:08:49.512197 493 leader_election.cc:288] T f496fabfd3f14a23b61368817358c88b P c542981ff53d422ba9a9eb6e0322a440 [CANDIDATE]: Term 1 election: Requested vote from peers 3640a584dd9d412facc2e8d05c476820 (127.5.83.2:35763), c6501645d9f54ec4b222ac740e56c479 (127.5.83.3:33673)
I20260808 04:08:49.512571 32668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f496fabfd3f14a23b61368817358c88b" candidate_uuid: "c542981ff53d422ba9a9eb6e0322a440" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3640a584dd9d412facc2e8d05c476820"
I20260808 04:08:49.512744 32668 raft_consensus.cc:3037] T f496fabfd3f14a23b61368817358c88b P 3640a584dd9d412facc2e8d05c476820 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:08:49.512992 386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f496fabfd3f14a23b61368817358c88b" candidate_uuid: "c542981ff53d422ba9a9eb6e0322a440" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6501645d9f54ec4b222ac740e56c479"
I20260808 04:08:49.513156 386 raft_consensus.cc:3037] T f496fabfd3f14a23b61368817358c88b P c6501645d9f54ec4b222ac740e56c479 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:08:49.515044 32668 raft_consensus.cc:2454] T f496fabfd3f14a23b61368817358c88b P 3640a584dd9d412facc2e8d05c476820 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c542981ff53d422ba9a9eb6e0322a440 in term 1.
I20260808 04:08:49.515401 32360 leader_election.cc:302] T f496fabfd3f14a23b61368817358c88b P c542981ff53d422ba9a9eb6e0322a440 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3640a584dd9d412facc2e8d05c476820, c542981ff53d422ba9a9eb6e0322a440; no voters:
I20260808 04:08:49.515564 386 raft_consensus.cc:2454] T f496fabfd3f14a23b61368817358c88b P c6501645d9f54ec4b222ac740e56c479 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c542981ff53d422ba9a9eb6e0322a440 in term 1.
I20260808 04:08:49.515672 493 raft_consensus.cc:2781] T f496fabfd3f14a23b61368817358c88b P c542981ff53d422ba9a9eb6e0322a440 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:08:49.516146 493 raft_consensus.cc:686] T f496fabfd3f14a23b61368817358c88b P c542981ff53d422ba9a9eb6e0322a440 [term 1 LEADER]: Becoming Leader. State: Replica: c542981ff53d422ba9a9eb6e0322a440, State: Running, Role: LEADER
I20260808 04:08:49.516641 493 consensus_queue.cc:237] T f496fabfd3f14a23b61368817358c88b P c542981ff53d422ba9a9eb6e0322a440 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3640a584dd9d412facc2e8d05c476820" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 35763 } } peers { permanent_uuid: "c6501645d9f54ec4b222ac740e56c479" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 33673 } } peers { permanent_uuid: "c542981ff53d422ba9a9eb6e0322a440" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 38463 } }
I20260808 04:08:49.521064 32253 catalog_manager.cc:5495] T f496fabfd3f14a23b61368817358c88b P c542981ff53d422ba9a9eb6e0322a440 reported cstate change: term changed from 0 to 1, leader changed from <none> to c542981ff53d422ba9a9eb6e0322a440 (127.5.83.1). New cstate: current_term: 1 leader_uuid: "c542981ff53d422ba9a9eb6e0322a440" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3640a584dd9d412facc2e8d05c476820" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 35763 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6501645d9f54ec4b222ac740e56c479" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 33673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c542981ff53d422ba9a9eb6e0322a440" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 38463 } health_report { overall_health: HEALTHY } } }
I20260808 04:08:49.558842 484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:49.553747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41722 (local address 127.5.83.1:38463)
0808 04:08:49.553916 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:49.553920 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:49.553989 (+ 69us) server_negotiation.cc:407] Connection header received
0808 04:08:49.554180 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:49.554185 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:49.554294 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:49.554467 (+ 173us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:08:49.554770 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:49.555510 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:49.556884 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:49.557277 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:49.557378 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:49.557897 (+ 519us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:08:49.558401 (+ 504us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:08:49.558532 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:49.558615 (+ 83us) server_negotiation.cc:299] Negotiation successful
0808 04:08:49.558652 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 04:08:49.566824 386 raft_consensus.cc:1261] T f496fabfd3f14a23b61368817358c88b P c6501645d9f54ec4b222ac740e56c479 [term 1 FOLLOWER]: Refusing update from remote peer c542981ff53d422ba9a9eb6e0322a440: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:08:49.566828 32668 raft_consensus.cc:1261] T f496fabfd3f14a23b61368817358c88b P 3640a584dd9d412facc2e8d05c476820 [term 1 FOLLOWER]: Refusing update from remote peer c542981ff53d422ba9a9eb6e0322a440: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:08:49.567823 500 consensus_queue.cc:1035] T f496fabfd3f14a23b61368817358c88b P c542981ff53d422ba9a9eb6e0322a440 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3640a584dd9d412facc2e8d05c476820" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 35763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:08:49.568166 493 consensus_queue.cc:1035] T f496fabfd3f14a23b61368817358c88b P c542981ff53d422ba9a9eb6e0322a440 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6501645d9f54ec4b222ac740e56c479" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 33673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:08:49.581475 503 mvcc.cc:205] Tried to move back new op lower bound from 7316120082695282688 to 7316120082503950336. Current Snapshot: MvccSnapshot[applied={T|T < 7316120082695282688}]
I20260808 04:08:49.585992 504 mvcc.cc:205] Tried to move back new op lower bound from 7316120082695282688 to 7316120082503950336. Current Snapshot: MvccSnapshot[applied={T|T < 7316120082695282688}]
I20260808 04:08:49.586611 505 mvcc.cc:205] Tried to move back new op lower bound from 7316120082695282688 to 7316120082503950336. Current Snapshot: MvccSnapshot[applied={T|T < 7316120082695282688}]
I20260808 04:08:49.596975 32253 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53618:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:08:49.597287 32253 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53618:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:08:49.601984 32253 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5160605195f149b1bdbd8ca5436050f9: Sending DeleteTablet for 3 replicas of tablet f496fabfd3f14a23b61368817358c88b
I20260808 04:08:49.603420 361 tablet_service.cc:1514] Processing DeleteTablet for tablet f496fabfd3f14a23b61368817358c88b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:08:49 UTC) from {username='kudu', principal='kudu/127.5.83.62@KRBTEST.COM'} at 127.0.0.1:60998
I20260808 04:08:49.604728 32635 tablet_service.cc:1514] Processing DeleteTablet for tablet f496fabfd3f14a23b61368817358c88b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:08:49 UTC) from {username='kudu', principal='kudu/127.5.83.62@KRBTEST.COM'} at 127.0.0.1:38336
I20260808 04:08:49.604703 32636 tablet_service.cc:1514] Processing DeleteTablet for tablet f496fabfd3f14a23b61368817358c88b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:08:49 UTC) from {username='kudu', principal='kudu/127.5.83.62@KRBTEST.COM'} at 127.0.0.1:38336
I20260808 04:08:49.605098 32428 tablet_service.cc:1514] Processing DeleteTablet for tablet f496fabfd3f14a23b61368817358c88b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:08:49 UTC) from {username='kudu', principal='kudu/127.5.83.62@KRBTEST.COM'} at 127.0.0.1:41708
I20260808 04:08:49.605892 32427 tablet_service.cc:1514] Processing DeleteTablet for tablet f496fabfd3f14a23b61368817358c88b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:08:49 UTC) from {username='kudu', principal='kudu/127.5.83.62@KRBTEST.COM'} at 127.0.0.1:41708
I20260808 04:08:49.607146 360 tablet_service.cc:1514] Processing DeleteTablet for tablet f496fabfd3f14a23b61368817358c88b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:08:49 UTC) from {username='kudu', principal='kudu/127.5.83.62@KRBTEST.COM'} at 127.0.0.1:60998
I20260808 04:08:49.608023 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 32339
I20260808 04:08:49.616317 519 tablet_replica.cc:330] T f496fabfd3f14a23b61368817358c88b P c542981ff53d422ba9a9eb6e0322a440: stopping tablet replica
I20260808 04:08:49.617393 519 raft_consensus.cc:2229] T f496fabfd3f14a23b61368817358c88b P c542981ff53d422ba9a9eb6e0322a440 [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:08:49.618687 519 raft_consensus.cc:2258] T f496fabfd3f14a23b61368817358c88b P c542981ff53d422ba9a9eb6e0322a440 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:08:49.622186 519 ts_tablet_manager.cc:1905] T f496fabfd3f14a23b61368817358c88b P c542981ff53d422ba9a9eb6e0322a440: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:08:49.626037 517 tablet_replica.cc:330] T f496fabfd3f14a23b61368817358c88b P c6501645d9f54ec4b222ac740e56c479: stopping tablet replica
I20260808 04:08:49.626305 518 tablet_replica.cc:330] T f496fabfd3f14a23b61368817358c88b P 3640a584dd9d412facc2e8d05c476820: stopping tablet replica
I20260808 04:08:49.626457 517 raft_consensus.cc:2229] T f496fabfd3f14a23b61368817358c88b P c6501645d9f54ec4b222ac740e56c479 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:08:49.626776 518 raft_consensus.cc:2229] T f496fabfd3f14a23b61368817358c88b P 3640a584dd9d412facc2e8d05c476820 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:08:49.627331 518 raft_consensus.cc:2258] T f496fabfd3f14a23b61368817358c88b P 3640a584dd9d412facc2e8d05c476820 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:08:49.627760 517 raft_consensus.cc:2258] T f496fabfd3f14a23b61368817358c88b P c6501645d9f54ec4b222ac740e56c479 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:08:49.629616 518 ts_tablet_manager.cc:1905] T f496fabfd3f14a23b61368817358c88b P 3640a584dd9d412facc2e8d05c476820: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:08:49.631278 517 ts_tablet_manager.cc:1905] T f496fabfd3f14a23b61368817358c88b P c6501645d9f54ec4b222ac740e56c479: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:08:49.632817 519 ts_tablet_manager.cc:1918] T f496fabfd3f14a23b61368817358c88b P c542981ff53d422ba9a9eb6e0322a440: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:08:49.633077 519 log.cc:1196] T f496fabfd3f14a23b61368817358c88b P c542981ff53d422ba9a9eb6e0322a440: Deleting WAL directory at /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/minicluster-data/ts-0/wal/wals/f496fabfd3f14a23b61368817358c88b
I20260808 04:08:49.633685 519 ts_tablet_manager.cc:1939] T f496fabfd3f14a23b61368817358c88b P c542981ff53d422ba9a9eb6e0322a440: Deleting consensus metadata
I20260808 04:08:49.635602 32234 catalog_manager.cc:4841] TS c542981ff53d422ba9a9eb6e0322a440 (127.5.83.1:38463): tablet f496fabfd3f14a23b61368817358c88b (table test-table [id=938b45f2307548d5b6bed308bedb7c46]) successfully deleted
I20260808 04:08:49.636737 518 ts_tablet_manager.cc:1918] T f496fabfd3f14a23b61368817358c88b P 3640a584dd9d412facc2e8d05c476820: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260808 04:08:49.636799 32234 catalog_manager.cc:4808] TS c542981ff53d422ba9a9eb6e0322a440 (127.5.83.1:38463): delete failed for tablet f496fabfd3f14a23b61368817358c88b because the tablet was not found. No further retry: Not found: Tablet not found: f496fabfd3f14a23b61368817358c88b
I20260808 04:08:49.636955 518 log.cc:1196] T f496fabfd3f14a23b61368817358c88b P 3640a584dd9d412facc2e8d05c476820: Deleting WAL directory at /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/minicluster-data/ts-1/wal/wals/f496fabfd3f14a23b61368817358c88b
I20260808 04:08:49.637527 518 ts_tablet_manager.cc:1939] T f496fabfd3f14a23b61368817358c88b P 3640a584dd9d412facc2e8d05c476820: Deleting consensus metadata
I20260808 04:08:49.638902 517 ts_tablet_manager.cc:1918] T f496fabfd3f14a23b61368817358c88b P c6501645d9f54ec4b222ac740e56c479: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:08:49.639050 517 log.cc:1196] T f496fabfd3f14a23b61368817358c88b P c6501645d9f54ec4b222ac740e56c479: Deleting WAL directory at /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161826605655-5452-0/minicluster-data/ts-2/wal/wals/f496fabfd3f14a23b61368817358c88b
I20260808 04:08:49.639374 32228 catalog_manager.cc:4841] TS 3640a584dd9d412facc2e8d05c476820 (127.5.83.2:35763): tablet f496fabfd3f14a23b61368817358c88b (table test-table [id=938b45f2307548d5b6bed308bedb7c46]) successfully deleted
I20260808 04:08:49.639559 517 ts_tablet_manager.cc:1939] T f496fabfd3f14a23b61368817358c88b P c6501645d9f54ec4b222ac740e56c479: Deleting consensus metadata
W20260808 04:08:49.640230 32228 catalog_manager.cc:4808] TS 3640a584dd9d412facc2e8d05c476820 (127.5.83.2:35763): delete failed for tablet f496fabfd3f14a23b61368817358c88b because the tablet was not found. No further retry: Not found: Tablet not found: f496fabfd3f14a23b61368817358c88b
I20260808 04:08:49.641142 32228 catalog_manager.cc:4841] TS c6501645d9f54ec4b222ac740e56c479 (127.5.83.3:33673): tablet f496fabfd3f14a23b61368817358c88b (table test-table [id=938b45f2307548d5b6bed308bedb7c46]) successfully deleted
W20260808 04:08:49.641474 32228 catalog_manager.cc:4808] TS c6501645d9f54ec4b222ac740e56c479 (127.5.83.3:33673): delete failed for tablet f496fabfd3f14a23b61368817358c88b because the tablet was not found. No further retry: Not found: Tablet not found: f496fabfd3f14a23b61368817358c88b
Aug 08 04:08:49 dist-test-slave-1l3l krb5kdc[32190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.1@KRBTEST.COM
I20260808 04:08:49.650431 484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:49.609700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41734 (local address 127.5.83.1:38463)
0808 04:08:49.609867 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:49.609874 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:49.623211 (+ 13337us) server_negotiation.cc:407] Connection header received
0808 04:08:49.623677 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:49.623683 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:49.623793 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:49.623865 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:08:49.626679 (+ 2814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:49.628056 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:49.631490 (+ 3434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:49.632233 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:49.632323 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:49.644354 (+ 12031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:08:49.644367 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:08:49.644399 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:08:49.644435 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:08:49.647428 (+ 2993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:08:49.648373 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:08:49.648397 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:08:49.648399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:08:49.648559 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:08:49.648980 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:08:49.648983 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:08:49.648985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:08:49.649340 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:08:49.649591 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:49.649956 (+ 365us) server_negotiation.cc:299] Negotiation successful
0808 04:08:49.650141 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 04:08:49.651232 32524 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:08:49.751674 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 32339
I20260808 04:08:49.775120 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 32540
Aug 08 04:08:49 dist-test-slave-1l3l krb5kdc[32190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.2@KRBTEST.COM
I20260808 04:08:49.789633 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:49.775945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38352 (local address 127.5.83.2:35763)
0808 04:08:49.776139 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:49.776144 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:49.776163 (+ 19us) server_negotiation.cc:407] Connection header received
0808 04:08:49.776284 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:49.776288 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:49.776427 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:49.776516 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:08:49.777854 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:49.778673 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:49.779991 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:49.780417 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:49.780513 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:49.784546 (+ 4033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:08:49.784561 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:08:49.784577 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:08:49.784608 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:08:49.787177 (+ 2569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:08:49.787873 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:08:49.787882 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:08:49.787884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:08:49.788005 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:08:49.788354 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:08:49.788357 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:08:49.788359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:08:49.788707 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:08:49.788927 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:49.789114 (+ 187us) server_negotiation.cc:299] Negotiation successful
0808 04:08:49.789285 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260808 04:08:49.790274 32724 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:08:49.877614 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 32540
I20260808 04:08:49.897205 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 32736
Aug 08 04:08:49 dist-test-slave-1l3l krb5kdc[32190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.3@KRBTEST.COM
I20260808 04:08:49.911818 486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:49.898039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32774 (local address 127.5.83.3:33673)
0808 04:08:49.898219 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:49.898225 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:49.898246 (+ 21us) server_negotiation.cc:407] Connection header received
0808 04:08:49.898459 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:49.898463 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:49.898533 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:49.898624 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:08:49.899940 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:49.900662 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:49.901955 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:49.902362 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:49.902476 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:49.906901 (+ 4425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:08:49.906917 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:08:49.906933 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:08:49.906974 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:08:49.909497 (+ 2523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:08:49.910080 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:08:49.910086 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:08:49.910088 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:08:49.910231 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:08:49.910553 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:08:49.910557 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:08:49.910560 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:08:49.910911 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:08:49.911126 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:49.911340 (+ 214us) server_negotiation.cc:299] Negotiation successful
0808 04:08:49.911507 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 04:08:49.912459 454 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:08:49.997939 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 32736
I20260808 04:08:50.018945 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 32207
I20260808 04:08:50.020028 32293 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:08:50.128119 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 32207
2026-08-08T04:08:50Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4427 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-08T04:08:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:08:50Z Disabled control of system clock
I20260808 04:08:50.210772 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161826605655-5452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161826605655-5452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161826605655-5452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:38461
--webserver_interface=127.5.83.62
--webserver_port=0
--builtin_ntp_servers=127.5.83.20:43367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.62:38461
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 04:08:50.526996 532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:08:50.527259 532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:08:50.527375 532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:08:50.534521 532 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:08:50.534615 532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:08:50.534677 532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:08:50.534751 532 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:08:50.534811 532 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:08:50.543637 532 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:43367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161826605655-5452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161826605655-5452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.62:38461
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:38461
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161826605655-5452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:08:50.544816 532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:08:50.546520 532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:08:50.557410 544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:08:50.557366 546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:08:50.557832 540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:08:50.558421 532 server_base.cc:1034] running on GCE node
I20260808 04:08:50.559219 532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:08:50.560523 532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:08:50.561725 532 hybrid_clock.cc:648] HybridClock initialized: now 1786162130561687 us; error 35 us; skew 500 ppm
I20260808 04:08:50.562038 532 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 04:08:50.564188 532 webserver.cc:460] Webserver started at http://127.5.83.62:36417/ using document root <none> and password file <none>
I20260808 04:08:50.564813 532 fs_manager.cc:359] Metadata directory not provided
I20260808 04:08:50.564916 532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:08:50.565222 532 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:08:50.568033 532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161826605655-5452-0/minicluster-data/master-0/data/instance:
uuid: "a3f52bb84d5241d89f64ac6e9bfd1b65"
format_stamp: "Formatted at 2026-08-08 04:08:50 on dist-test-slave-1l3l"
server_key: "32211e51267bb8438e91746cc2f35e49"
server_key_iv: "e8afdb677ed9a04585a4e048c80c6ccc"
server_key_version: "encryptionkey@0"
I20260808 04:08:50.568791 532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161826605655-5452-0/minicluster-data/master-0/wal/instance:
uuid: "a3f52bb84d5241d89f64ac6e9bfd1b65"
format_stamp: "Formatted at 2026-08-08 04:08:50 on dist-test-slave-1l3l"
server_key: "32211e51267bb8438e91746cc2f35e49"
server_key_iv: "e8afdb677ed9a04585a4e048c80c6ccc"
server_key_version: "encryptionkey@0"
I20260808 04:08:50.573508 532 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260808 04:08:50.576679 553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:08:50.577818 532 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 04:08:50.577976 532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161826605655-5452-0/minicluster-data/master-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161826605655-5452-0/minicluster-data/master-0/wal
uuid: "a3f52bb84d5241d89f64ac6e9bfd1b65"
format_stamp: "Formatted at 2026-08-08 04:08:50 on dist-test-slave-1l3l"
server_key: "32211e51267bb8438e91746cc2f35e49"
server_key_iv: "e8afdb677ed9a04585a4e048c80c6ccc"
server_key_version: "encryptionkey@0"
I20260808 04:08:50.578176 532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161826605655-5452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161826605655-5452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161826605655-5452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:08:50.597609 532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:08:50.598345 532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:08:50.598608 532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:08:50.614492 532 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.62:38461
I20260808 04:08:50.614464 624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.62:38461 every 8 connection(s)
I20260808 04:08:50.615790 532 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
I20260808 04:08:50.619240 627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:08:50.622237 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 532
I20260808 04:08:50.622426 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161826605655-5452-0/minicluster-data/master-0/wal/instance
I20260808 04:08:50.622956 5452 external_mini_cluster.cc:1428] Setting key 180b347b0c519269a4bb5e46e8d97463
I20260808 04:08:50.631062 627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3f52bb84d5241d89f64ac6e9bfd1b65: Bootstrap starting.
I20260808 04:08:50.634483 627 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3f52bb84d5241d89f64ac6e9bfd1b65: Neither blocks nor log segments found. Creating new log.
I20260808 04:08:50.635617 627 log.cc:824] T 00000000000000000000000000000000 P a3f52bb84d5241d89f64ac6e9bfd1b65: Log is configured to *not* fsync() on all Append() calls
I20260808 04:08:50.636132 630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:50.624833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44598 (local address 127.5.83.62:38461)
0808 04:08:50.625729 (+ 896us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:50.625739 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:50.625782 (+ 43us) server_negotiation.cc:407] Connection header received
0808 04:08:50.627379 (+ 1597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:50.627405 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:50.627890 (+ 485us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:50.628204 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:08:50.628784 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:50.630032 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:50.631852 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:50.632520 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:50.632651 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:50.633863 (+ 1212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:08:50.633870 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:08:50.633883 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:08:50.634368 (+ 485us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:08:50.634481 (+ 113us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:08:50.634501 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:08:50.634602 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:50.634816 (+ 214us) server_negotiation.cc:299] Negotiation successful
0808 04:08:50.635112 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":308,"threads_started":1}
I20260808 04:08:50.638365 627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3f52bb84d5241d89f64ac6e9bfd1b65: No bootstrap required, opened a new log
I20260808 04:08:50.645972 627 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3f52bb84d5241d89f64ac6e9bfd1b65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3f52bb84d5241d89f64ac6e9bfd1b65" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 38461 } }
I20260808 04:08:50.646333 627 raft_consensus.cc:374] T 00000000000000000000000000000000 P a3f52bb84d5241d89f64ac6e9bfd1b65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:08:50.646435 627 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3f52bb84d5241d89f64ac6e9bfd1b65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3f52bb84d5241d89f64ac6e9bfd1b65, State: Initialized, Role: FOLLOWER
I20260808 04:08:50.646950 627 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3f52bb84d5241d89f64ac6e9bfd1b65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3f52bb84d5241d89f64ac6e9bfd1b65" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 38461 } }
I20260808 04:08:50.647166 627 raft_consensus.cc:388] T 00000000000000000000000000000000 P a3f52bb84d5241d89f64ac6e9bfd1b65 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:08:50.647295 627 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3f52bb84d5241d89f64ac6e9bfd1b65 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:08:50.647445 627 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3f52bb84d5241d89f64ac6e9bfd1b65 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:08:50.649674 627 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3f52bb84d5241d89f64ac6e9bfd1b65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3f52bb84d5241d89f64ac6e9bfd1b65" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 38461 } }
I20260808 04:08:50.650134 627 leader_election.cc:302] T 00000000000000000000000000000000 P a3f52bb84d5241d89f64ac6e9bfd1b65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a3f52bb84d5241d89f64ac6e9bfd1b65; no voters:
I20260808 04:08:50.650792 627 leader_election.cc:288] T 00000000000000000000000000000000 P a3f52bb84d5241d89f64ac6e9bfd1b65 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:08:50.650906 632 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a3f52bb84d5241d89f64ac6e9bfd1b65 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:08:50.651939 632 raft_consensus.cc:686] T 00000000000000000000000000000000 P a3f52bb84d5241d89f64ac6e9bfd1b65 [term 1 LEADER]: Becoming Leader. State: Replica: a3f52bb84d5241d89f64ac6e9bfd1b65, State: Running, Role: LEADER
I20260808 04:08:50.652391 632 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3f52bb84d5241d89f64ac6e9bfd1b65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3f52bb84d5241d89f64ac6e9bfd1b65" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 38461 } }
I20260808 04:08:50.652895 627 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3f52bb84d5241d89f64ac6e9bfd1b65 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:08:50.655019 633 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3f52bb84d5241d89f64ac6e9bfd1b65 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a3f52bb84d5241d89f64ac6e9bfd1b65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3f52bb84d5241d89f64ac6e9bfd1b65" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 38461 } } }
I20260808 04:08:50.655259 634 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3f52bb84d5241d89f64ac6e9bfd1b65 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3f52bb84d5241d89f64ac6e9bfd1b65. Latest consensus state: current_term: 1 leader_uuid: "a3f52bb84d5241d89f64ac6e9bfd1b65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3f52bb84d5241d89f64ac6e9bfd1b65" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 38461 } } }
I20260808 04:08:50.655565 633 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3f52bb84d5241d89f64ac6e9bfd1b65 [sys.catalog]: This master's current role is: LEADER
I20260808 04:08:50.655618 634 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3f52bb84d5241d89f64ac6e9bfd1b65 [sys.catalog]: This master's current role is: LEADER
I20260808 04:08:50.659350 646 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:08:50.664512 646 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:08:50.672750 646 catalog_manager.cc:1347] Generated new cluster ID: 069991883a754c168cebf54cb9f35edb
I20260808 04:08:50.672850 646 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:08:50.693945 646 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:08:50.695135 646 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:08:50.705214 646 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a3f52bb84d5241d89f64ac6e9bfd1b65: Generated new TSK 0
I20260808 04:08:50.705950 646 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:08:50.717682 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161826605655-5452-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161826605655-5452-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161826605655-5452-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.1:0
--local_ip_for_outbound_sockets=127.5.83.1
--webserver_interface=127.5.83.1
--webserver_port=0
--tserver_master_addrs=127.5.83.62:38461
--builtin_ntp_servers=127.5.83.20:43367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 04:08:51.047281 658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:08:51.047564 658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:08:51.047679 658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:08:51.055178 658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:08:51.055287 658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:08:51.055478 658 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.1
I20260808 04:08:51.064926 658 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:43367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161826605655-5452-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161826605655-5452-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.83.1
--webserver_port=0
--tserver_master_addrs=127.5.83.62:38461
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.1
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161826605655-5452-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:08:51.066241 658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:08:51.068058 658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:08:51.080276 669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:08:51.080291 667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:08:51.080299 666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:08:51.081269 658 server_base.cc:1034] running on GCE node
I20260808 04:08:51.081835 658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:08:51.082784 658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:08:51.083988 658 hybrid_clock.cc:648] HybridClock initialized: now 1786162131083944 us; error 35 us; skew 500 ppm
I20260808 04:08:51.084282 658 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 04:08:51.086429 658 webserver.cc:460] Webserver started at http://127.5.83.1:37973/ using document root <none> and password file <none>
I20260808 04:08:51.087070 658 fs_manager.cc:359] Metadata directory not provided
I20260808 04:08:51.087173 658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:08:51.087493 658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:08:51.090286 658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161826605655-5452-0/minicluster-data/ts-0/data/instance:
uuid: "55739b89d3e64a528951c00448bc03bc"
format_stamp: "Formatted at 2026-08-08 04:08:51 on dist-test-slave-1l3l"
server_key: "34198cf4bdb60d56f70a37cbc5a90f95"
server_key_iv: "1b29d787993ea029a4d1257b705ae31f"
server_key_version: "encryptionkey@0"
I20260808 04:08:51.091074 658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161826605655-5452-0/minicluster-data/ts-0/wal/instance:
uuid: "55739b89d3e64a528951c00448bc03bc"
format_stamp: "Formatted at 2026-08-08 04:08:51 on dist-test-slave-1l3l"
server_key: "34198cf4bdb60d56f70a37cbc5a90f95"
server_key_iv: "1b29d787993ea029a4d1257b705ae31f"
server_key_version: "encryptionkey@0"
I20260808 04:08:51.095980 658 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260808 04:08:51.099221 680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:08:51.100469 658 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:08:51.100617 658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161826605655-5452-0/minicluster-data/ts-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161826605655-5452-0/minicluster-data/ts-0/wal
uuid: "55739b89d3e64a528951c00448bc03bc"
format_stamp: "Formatted at 2026-08-08 04:08:51 on dist-test-slave-1l3l"
server_key: "34198cf4bdb60d56f70a37cbc5a90f95"
server_key_iv: "1b29d787993ea029a4d1257b705ae31f"
server_key_version: "encryptionkey@0"
I20260808 04:08:51.100807 658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161826605655-5452-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161826605655-5452-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161826605655-5452-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:08:51.116457 658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:08:51.117172 658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:08:51.117429 658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:08:51.118299 658 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:08:51.119669 658 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:08:51.119796 658 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:08:51.119903 658 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:08:51.119974 658 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:08:51.151088 658 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.1:36851
I20260808 04:08:51.151136 834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.1:36851 every 8 connection(s)
I20260808 04:08:51.152459 658 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
I20260808 04:08:51.157864 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 658
I20260808 04:08:51.158020 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161826605655-5452-0/minicluster-data/ts-0/wal/instance
I20260808 04:08:51.158439 5452 external_mini_cluster.cc:1428] Setting key 1e33a6de979c277cdd201de1ef8325bf
I20260808 04:08:51.161217 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161826605655-5452-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161826605655-5452-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161826605655-5452-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.2:0
--local_ip_for_outbound_sockets=127.5.83.2
--webserver_interface=127.5.83.2
--webserver_port=0
--tserver_master_addrs=127.5.83.62:38461
--builtin_ntp_servers=127.5.83.20:43367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 04:08:51.165000 838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:51.154989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.1:58727 (local address 127.5.83.62:38461)
0808 04:08:51.155436 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:51.155441 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:51.156592 (+ 1151us) server_negotiation.cc:407] Connection header received
0808 04:08:51.158414 (+ 1822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:51.158421 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:51.158501 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:51.158614 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:08:51.159970 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:51.160805 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:51.162381 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:51.162796 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:51.162907 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:51.164183 (+ 1276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:08:51.164189 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:08:51.164194 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:08:51.164213 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:08:51.164261 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:08:51.164272 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:08:51.164353 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:51.164562 (+ 209us) server_negotiation.cc:299] Negotiation successful
0808 04:08:51.164657 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":204,"threads_started":1}
I20260808 04:08:51.165915 842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:51.155483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:38461 (local address 127.5.83.1:58727)
0808 04:08:51.156444 (+ 961us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:51.156480 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0808 04:08:51.158156 (+ 1676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:08:51.158914 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:08:51.158925 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:08:51.159033 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:08:51.159783 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:51.159808 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:51.160964 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:51.160968 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:51.162237 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:51.162249 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:51.163008 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:51.163015 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:51.163234 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:51.163899 (+ 665us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:08:51.163902 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:08:51.163939 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:08:51.164010 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:08:51.164357 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:08:51.164368 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:08:51.164376 (+ 8us) client_negotiation.cc:769] Sending connection context
0808 04:08:51.164519 (+ 143us) client_negotiation.cc:240] Negotiation successful
0808 04:08:51.164898 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":346,"threads_started":1}
I20260808 04:08:51.167352 836 heartbeater.cc:347] Connected to a master server at 127.5.83.62:38461
I20260808 04:08:51.167619 836 heartbeater.cc:464] Registering TS with master...
I20260808 04:08:51.168391 836 heartbeater.cc:511] Master 127.5.83.62:38461 requested a full tablet report, sending...
I20260808 04:08:51.170552 575 ts_manager.cc:194] Registered new tserver with Master: 55739b89d3e64a528951c00448bc03bc (127.5.83.1:36851)
I20260808 04:08:51.172418 575 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.83.1:58727
W20260808 04:08:51.491492 843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:08:51.491803 843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:08:51.491933 843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:08:51.498924 843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:08:51.499020 843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:08:51.499161 843 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.2
I20260808 04:08:51.507984 843 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:43367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161826605655-5452-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161826605655-5452-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.83.2
--webserver_port=0
--tserver_master_addrs=127.5.83.62:38461
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.2
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161826605655-5452-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:08:51.509155 843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:08:51.510860 843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:08:51.523213 850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:08:51.523252 854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:08:51.523303 851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:08:51.529021 843 server_base.cc:1034] running on GCE node
I20260808 04:08:51.529567 843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:08:51.530480 843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:08:51.531677 843 hybrid_clock.cc:648] HybridClock initialized: now 1786162131531625 us; error 48 us; skew 500 ppm
I20260808 04:08:51.532008 843 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 04:08:51.534071 843 webserver.cc:460] Webserver started at http://127.5.83.2:40807/ using document root <none> and password file <none>
I20260808 04:08:51.534700 843 fs_manager.cc:359] Metadata directory not provided
I20260808 04:08:51.534793 843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:08:51.535115 843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:08:51.537894 843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161826605655-5452-0/minicluster-data/ts-1/data/instance:
uuid: "078aaad2a15b48f5bc0917eff7c92fb5"
format_stamp: "Formatted at 2026-08-08 04:08:51 on dist-test-slave-1l3l"
server_key: "b9f64f78760b7bfd99cc8385dc854e08"
server_key_iv: "02729314549db8dc232e449d2f5307eb"
server_key_version: "encryptionkey@0"
I20260808 04:08:51.538641 843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161826605655-5452-0/minicluster-data/ts-1/wal/instance:
uuid: "078aaad2a15b48f5bc0917eff7c92fb5"
format_stamp: "Formatted at 2026-08-08 04:08:51 on dist-test-slave-1l3l"
server_key: "b9f64f78760b7bfd99cc8385dc854e08"
server_key_iv: "02729314549db8dc232e449d2f5307eb"
server_key_version: "encryptionkey@0"
I20260808 04:08:51.543318 843 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260808 04:08:51.546496 862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:08:51.547636 843 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:08:51.547806 843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161826605655-5452-0/minicluster-data/ts-1/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161826605655-5452-0/minicluster-data/ts-1/wal
uuid: "078aaad2a15b48f5bc0917eff7c92fb5"
format_stamp: "Formatted at 2026-08-08 04:08:51 on dist-test-slave-1l3l"
server_key: "b9f64f78760b7bfd99cc8385dc854e08"
server_key_iv: "02729314549db8dc232e449d2f5307eb"
server_key_version: "encryptionkey@0"
I20260808 04:08:51.547993 843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161826605655-5452-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161826605655-5452-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161826605655-5452-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:08:51.563043 843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:08:51.563769 843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:08:51.564006 843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:08:51.564878 843 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:08:51.566200 843 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:08:51.566299 843 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:08:51.566406 843 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:08:51.566474 843 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:08:51.595683 843 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.2:39335
I20260808 04:08:51.595765 1035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.2:39335 every 8 connection(s)
I20260808 04:08:51.596940 843 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
I20260808 04:08:51.600459 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 843
I20260808 04:08:51.600646 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161826605655-5452-0/minicluster-data/ts-1/wal/instance
I20260808 04:08:51.601186 5452 external_mini_cluster.cc:1428] Setting key 93dc65525c2151d7b3e6a9aff6af6422
I20260808 04:08:51.604323 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161826605655-5452-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161826605655-5452-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161826605655-5452-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.3:0
--local_ip_for_outbound_sockets=127.5.83.3
--webserver_interface=127.5.83.3
--webserver_port=0
--tserver_master_addrs=127.5.83.62:38461
--builtin_ntp_servers=127.5.83.20:43367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 04:08:51.610239 838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:51.599338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.2:51939 (local address 127.5.83.62:38461)
0808 04:08:51.599524 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:51.599528 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:51.601118 (+ 1590us) server_negotiation.cc:407] Connection header received
0808 04:08:51.603244 (+ 2126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:51.603248 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:51.603311 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:51.603392 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:08:51.604877 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:51.605630 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:51.607693 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:51.608128 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:51.608221 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:51.609596 (+ 1375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:08:51.609601 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:08:51.609604 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:08:51.609622 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:08:51.609652 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:08:51.609663 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:08:51.609728 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:51.609935 (+ 207us) server_negotiation.cc:299] Negotiation successful
0808 04:08:51.610009 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260808 04:08:51.611470 1040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:51.599916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:38461 (local address 127.5.83.2:51939)
0808 04:08:51.600952 (+ 1036us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:51.601007 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0808 04:08:51.603004 (+ 1997us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:08:51.603647 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:08:51.603658 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:08:51.603795 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:08:51.604675 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:51.604700 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:51.605775 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:51.605781 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:51.607533 (+ 1752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:51.607547 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:51.608333 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:51.608340 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:51.608588 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:51.609294 (+ 706us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:08:51.609298 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:08:51.609343 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:08:51.609428 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:08:51.609744 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:08:51.609755 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:08:51.609762 (+ 7us) client_negotiation.cc:769] Sending connection context
0808 04:08:51.609933 (+ 171us) client_negotiation.cc:240] Negotiation successful
0808 04:08:51.610269 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":299,"threads_started":1}
I20260808 04:08:51.612787 1036 heartbeater.cc:347] Connected to a master server at 127.5.83.62:38461
I20260808 04:08:51.613078 1036 heartbeater.cc:464] Registering TS with master...
I20260808 04:08:51.613914 1036 heartbeater.cc:511] Master 127.5.83.62:38461 requested a full tablet report, sending...
I20260808 04:08:51.615419 575 ts_manager.cc:194] Registered new tserver with Master: 078aaad2a15b48f5bc0917eff7c92fb5 (127.5.83.2:39335)
I20260808 04:08:51.616566 575 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.83.2:51939
W20260808 04:08:51.928726 1042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:08:51.928992 1042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:08:51.929114 1042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:08:51.936194 1042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:08:51.936281 1042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:08:51.936424 1042 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.3
I20260808 04:08:51.945128 1042 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:43367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161826605655-5452-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161826605655-5452-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.83.3
--webserver_port=0
--tserver_master_addrs=127.5.83.62:38461
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.3
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161826605655-5452-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:08:51.946300 1042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:08:51.948035 1042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:08:51.960541 1049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:08:51.960505 1054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:08:51.960573 1048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:08:51.961761 1042 server_base.cc:1034] running on GCE node
I20260808 04:08:51.962350 1042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:08:51.963218 1042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:08:51.964444 1042 hybrid_clock.cc:648] HybridClock initialized: now 1786162131964373 us; error 50 us; skew 500 ppm
I20260808 04:08:51.964756 1042 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 04:08:51.966799 1042 webserver.cc:460] Webserver started at http://127.5.83.3:36869/ using document root <none> and password file <none>
I20260808 04:08:51.967432 1042 fs_manager.cc:359] Metadata directory not provided
I20260808 04:08:51.967535 1042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:08:51.967864 1042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:08:51.970561 1042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161826605655-5452-0/minicluster-data/ts-2/data/instance:
uuid: "190405558faf4fd292c52e057b794b33"
format_stamp: "Formatted at 2026-08-08 04:08:51 on dist-test-slave-1l3l"
server_key: "5f604594b86c4d8f631498b27fe4d9a3"
server_key_iv: "75f6460144a57f595c592e2e9b63758b"
server_key_version: "encryptionkey@0"
I20260808 04:08:51.971314 1042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161826605655-5452-0/minicluster-data/ts-2/wal/instance:
uuid: "190405558faf4fd292c52e057b794b33"
format_stamp: "Formatted at 2026-08-08 04:08:51 on dist-test-slave-1l3l"
server_key: "5f604594b86c4d8f631498b27fe4d9a3"
server_key_iv: "75f6460144a57f595c592e2e9b63758b"
server_key_version: "encryptionkey@0"
I20260808 04:08:51.976053 1042 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260808 04:08:51.979164 1061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:08:51.980383 1042 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:08:51.980546 1042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161826605655-5452-0/minicluster-data/ts-2/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161826605655-5452-0/minicluster-data/ts-2/wal
uuid: "190405558faf4fd292c52e057b794b33"
format_stamp: "Formatted at 2026-08-08 04:08:51 on dist-test-slave-1l3l"
server_key: "5f604594b86c4d8f631498b27fe4d9a3"
server_key_iv: "75f6460144a57f595c592e2e9b63758b"
server_key_version: "encryptionkey@0"
I20260808 04:08:51.980733 1042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161826605655-5452-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161826605655-5452-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161826605655-5452-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:08:52.002031 1042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:08:52.002744 1042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:08:52.002992 1042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:08:52.003930 1042 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:08:52.005390 1042 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:08:52.005499 1042 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:08:52.005604 1042 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:08:52.005681 1042 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:08:52.036052 1042 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.3:38091
I20260808 04:08:52.036120 1208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.3:38091 every 8 connection(s)
I20260808 04:08:52.037297 1042 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
I20260808 04:08:52.045729 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 1042
I20260808 04:08:52.045902 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161826605655-5452-0/minicluster-data/ts-2/wal/instance
I20260808 04:08:52.046358 5452 external_mini_cluster.cc:1428] Setting key 754a6fbe924667a5493eb29855cef389
I20260808 04:08:52.049000 838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:52.039637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.3:56447 (local address 127.5.83.62:38461)
0808 04:08:52.039886 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:52.039892 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:52.041165 (+ 1273us) server_negotiation.cc:407] Connection header received
0808 04:08:52.042880 (+ 1715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:52.042885 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:52.042961 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:52.043048 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:08:52.044106 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:52.044840 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:52.046423 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:52.046871 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:52.046981 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:52.048251 (+ 1270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:08:52.048255 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:08:52.048259 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:08:52.048274 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:08:52.048306 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:08:52.048317 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:08:52.048383 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:52.048614 (+ 231us) server_negotiation.cc:299] Negotiation successful
0808 04:08:52.048723 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260808 04:08:52.049928 1211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:52.040171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:38461 (local address 127.5.83.3:56447)
0808 04:08:52.041000 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:52.041051 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 04:08:52.042627 (+ 1576us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:08:52.043278 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:08:52.043286 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:08:52.043369 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:08:52.043943 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:52.043961 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:52.044985 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:52.044989 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:52.046289 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:52.046302 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:52.047088 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:52.047096 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:52.047299 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:52.047970 (+ 671us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:08:52.047972 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:08:52.048009 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:08:52.048098 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:08:52.048416 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:08:52.048426 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:08:52.048433 (+ 7us) client_negotiation.cc:769] Sending connection context
0808 04:08:52.048604 (+ 171us) client_negotiation.cc:240] Negotiation successful
0808 04:08:52.048958 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":267,"threads_started":1}
I20260808 04:08:52.051050 1209 heartbeater.cc:347] Connected to a master server at 127.5.83.62:38461
I20260808 04:08:52.051324 1209 heartbeater.cc:464] Registering TS with master...
I20260808 04:08:52.052067 1209 heartbeater.cc:511] Master 127.5.83.62:38461 requested a full tablet report, sending...
I20260808 04:08:52.053359 575 ts_manager.cc:194] Registered new tserver with Master: 190405558faf4fd292c52e057b794b33 (127.5.83.3:38091)
I20260808 04:08:52.054184 575 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.83.3:56447
I20260808 04:08:52.061239 5452 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:08:52.070461 838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:52.065128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44604 (local address 127.5.83.62:38461)
0808 04:08:52.065275 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:52.065280 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:52.065723 (+ 443us) server_negotiation.cc:407] Connection header received
0808 04:08:52.065848 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:52.065851 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:52.065923 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:52.066002 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:08:52.066393 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:52.067084 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:52.068423 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:52.068822 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:52.068908 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:52.069880 (+ 972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:08:52.069885 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:08:52.069888 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:08:52.069907 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:08:52.069937 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:08:52.069948 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:08:52.070053 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:52.070181 (+ 128us) server_negotiation.cc:299] Negotiation successful
0808 04:08:52.070222 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 04:08:52.073933 575 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44604:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:08:52.080555 575 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:08:52.101373 1230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:52.092836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.3:38091 (local address 127.0.0.1:38710)
0808 04:08:52.093397 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:52.093414 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:08:52.093518 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:08:52.094638 (+ 1120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:08:52.094643 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:08:52.094669 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:08:52.095109 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:52.095122 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:52.097096 (+ 1974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:52.097105 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:52.099190 (+ 2085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:52.099204 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:52.100498 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:52.100504 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:52.100702 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:52.100761 (+ 59us) client_negotiation.cc:769] Sending connection context
0808 04:08:52.100893 (+ 132us) client_negotiation.cc:240] Negotiation successful
0808 04:08:52.101064 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":296,"threads_started":1}
I20260808 04:08:52.101485 1232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:52.093263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38710 (local address 127.5.83.3:38091)
0808 04:08:52.094066 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:52.094083 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:52.094115 (+ 32us) server_negotiation.cc:407] Connection header received
0808 04:08:52.094204 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:52.094211 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:52.094423 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:52.095323 (+ 900us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:08:52.095615 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:52.096942 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:52.099408 (+ 2466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:52.100324 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:52.100473 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:52.100742 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:52.100948 (+ 206us) server_negotiation.cc:299] Negotiation successful
0808 04:08:52.101113 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":645,"thread_start_us":457,"threads_started":1}
I20260808 04:08:52.101922 1227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:52.091618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.2:39335 (local address 127.0.0.1:49808)
0808 04:08:52.092361 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:52.092412 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 04:08:52.092578 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:08:52.093677 (+ 1099us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:08:52.093685 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:08:52.093713 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:08:52.094160 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:52.094173 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:52.096792 (+ 2619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:52.096797 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:52.098873 (+ 2076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:52.098889 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:52.100730 (+ 1841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:52.100736 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:52.100882 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:52.100913 (+ 31us) client_negotiation.cc:769] Sending connection context
0808 04:08:52.100964 (+ 51us) client_negotiation.cc:240] Negotiation successful
0808 04:08:52.101063 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"mutex_wait_us":26,"thread_start_us":346,"threads_started":1}
I20260808 04:08:52.102018 1228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:52.092001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49808 (local address 127.5.83.2:39335)
0808 04:08:52.093041 (+ 1040us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:52.093050 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:52.093082 (+ 32us) server_negotiation.cc:407] Connection header received
0808 04:08:52.093184 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:52.093192 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:52.093446 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:52.094368 (+ 922us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:08:52.095206 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:52.096653 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:52.099111 (+ 2458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:52.100528 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:52.101173 (+ 645us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:52.101429 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:52.101508 (+ 79us) server_negotiation.cc:299] Negotiation successful
0808 04:08:52.101654 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":887,"thread_start_us":756,"threads_started":1}
I20260808 04:08:52.103574 1233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:52.094001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:36851 (local address 127.0.0.1:59148)
0808 04:08:52.094508 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:52.094525 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:08:52.094636 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:08:52.095841 (+ 1205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:08:52.095846 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:08:52.095872 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:08:52.096216 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:52.096228 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:52.098110 (+ 1882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:52.098115 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:52.100194 (+ 2079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:52.100209 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:52.102894 (+ 2685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:52.102901 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:52.103074 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:52.103104 (+ 30us) client_negotiation.cc:769] Sending connection context
0808 04:08:52.103199 (+ 95us) client_negotiation.cc:240] Negotiation successful
0808 04:08:52.103314 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":695,"threads_started":1}
I20260808 04:08:52.104372 1234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:52.094505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59148 (local address 127.5.83.1:36851)
0808 04:08:52.095206 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:52.095212 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:52.095237 (+ 25us) server_negotiation.cc:407] Connection header received
0808 04:08:52.095360 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:52.095367 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:52.095572 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:52.095848 (+ 276us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:08:52.096419 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:52.097936 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:52.100393 (+ 2457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:52.102387 (+ 1994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:52.102505 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:52.102724 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:52.103872 (+ 1148us) server_negotiation.cc:299] Negotiation successful
0808 04:08:52.104043 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":458,"threads_started":1}
I20260808 04:08:52.105856 1129 tablet_service.cc:1467] Processing CreateTablet for tablet 92d6d4818e6a4b6587573bb75bca5dbf (DEFAULT_TABLE table=test-table [id=dd581f527bac45b29ed6cad205661c49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:08:52.105943 940 tablet_service.cc:1467] Processing CreateTablet for tablet 92d6d4818e6a4b6587573bb75bca5dbf (DEFAULT_TABLE table=test-table [id=dd581f527bac45b29ed6cad205661c49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:08:52.107108 1129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92d6d4818e6a4b6587573bb75bca5dbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:08:52.107344 940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92d6d4818e6a4b6587573bb75bca5dbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:08:52.107728 749 tablet_service.cc:1467] Processing CreateTablet for tablet 92d6d4818e6a4b6587573bb75bca5dbf (DEFAULT_TABLE table=test-table [id=dd581f527bac45b29ed6cad205661c49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:08:52.108963 749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92d6d4818e6a4b6587573bb75bca5dbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:08:52.117883 1235 tablet_bootstrap.cc:492] T 92d6d4818e6a4b6587573bb75bca5dbf P 190405558faf4fd292c52e057b794b33: Bootstrap starting.
I20260808 04:08:52.120961 1236 tablet_bootstrap.cc:492] T 92d6d4818e6a4b6587573bb75bca5dbf P 078aaad2a15b48f5bc0917eff7c92fb5: Bootstrap starting.
I20260808 04:08:52.121685 1235 tablet_bootstrap.cc:654] T 92d6d4818e6a4b6587573bb75bca5dbf P 190405558faf4fd292c52e057b794b33: Neither blocks nor log segments found. Creating new log.
I20260808 04:08:52.122546 1237 tablet_bootstrap.cc:492] T 92d6d4818e6a4b6587573bb75bca5dbf P 55739b89d3e64a528951c00448bc03bc: Bootstrap starting.
I20260808 04:08:52.123415 1235 log.cc:824] T 92d6d4818e6a4b6587573bb75bca5dbf P 190405558faf4fd292c52e057b794b33: Log is configured to *not* fsync() on all Append() calls
I20260808 04:08:52.124580 1236 tablet_bootstrap.cc:654] T 92d6d4818e6a4b6587573bb75bca5dbf P 078aaad2a15b48f5bc0917eff7c92fb5: Neither blocks nor log segments found. Creating new log.
I20260808 04:08:52.125415 1237 tablet_bootstrap.cc:654] T 92d6d4818e6a4b6587573bb75bca5dbf P 55739b89d3e64a528951c00448bc03bc: Neither blocks nor log segments found. Creating new log.
I20260808 04:08:52.125866 1236 log.cc:824] T 92d6d4818e6a4b6587573bb75bca5dbf P 078aaad2a15b48f5bc0917eff7c92fb5: Log is configured to *not* fsync() on all Append() calls
I20260808 04:08:52.126296 1237 log.cc:824] T 92d6d4818e6a4b6587573bb75bca5dbf P 55739b89d3e64a528951c00448bc03bc: Log is configured to *not* fsync() on all Append() calls
I20260808 04:08:52.126643 1235 tablet_bootstrap.cc:492] T 92d6d4818e6a4b6587573bb75bca5dbf P 190405558faf4fd292c52e057b794b33: No bootstrap required, opened a new log
I20260808 04:08:52.126897 1235 ts_tablet_manager.cc:1397] T 92d6d4818e6a4b6587573bb75bca5dbf P 190405558faf4fd292c52e057b794b33: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260808 04:08:52.128676 1236 tablet_bootstrap.cc:492] T 92d6d4818e6a4b6587573bb75bca5dbf P 078aaad2a15b48f5bc0917eff7c92fb5: No bootstrap required, opened a new log
I20260808 04:08:52.128942 1236 ts_tablet_manager.cc:1397] T 92d6d4818e6a4b6587573bb75bca5dbf P 078aaad2a15b48f5bc0917eff7c92fb5: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20260808 04:08:52.129246 1237 tablet_bootstrap.cc:492] T 92d6d4818e6a4b6587573bb75bca5dbf P 55739b89d3e64a528951c00448bc03bc: No bootstrap required, opened a new log
I20260808 04:08:52.129490 1237 ts_tablet_manager.cc:1397] T 92d6d4818e6a4b6587573bb75bca5dbf P 55739b89d3e64a528951c00448bc03bc: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260808 04:08:52.134223 1235 raft_consensus.cc:348] T 92d6d4818e6a4b6587573bb75bca5dbf P 190405558faf4fd292c52e057b794b33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078aaad2a15b48f5bc0917eff7c92fb5" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 39335 } } peers { permanent_uuid: "190405558faf4fd292c52e057b794b33" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 38091 } } peers { permanent_uuid: "55739b89d3e64a528951c00448bc03bc" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 36851 } }
I20260808 04:08:52.134564 1235 raft_consensus.cc:374] T 92d6d4818e6a4b6587573bb75bca5dbf P 190405558faf4fd292c52e057b794b33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:08:52.134660 1235 raft_consensus.cc:729] T 92d6d4818e6a4b6587573bb75bca5dbf P 190405558faf4fd292c52e057b794b33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 190405558faf4fd292c52e057b794b33, State: Initialized, Role: FOLLOWER
I20260808 04:08:52.135228 1235 consensus_queue.cc:260] T 92d6d4818e6a4b6587573bb75bca5dbf P 190405558faf4fd292c52e057b794b33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078aaad2a15b48f5bc0917eff7c92fb5" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 39335 } } peers { permanent_uuid: "190405558faf4fd292c52e057b794b33" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 38091 } } peers { permanent_uuid: "55739b89d3e64a528951c00448bc03bc" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 36851 } }
I20260808 04:08:52.136257 1209 heartbeater.cc:503] Master 127.5.83.62:38461 was elected leader, sending a full tablet report...
I20260808 04:08:52.136901 1235 ts_tablet_manager.cc:1428] T 92d6d4818e6a4b6587573bb75bca5dbf P 190405558faf4fd292c52e057b794b33: Time spent starting tablet: real 0.010s user 0.008s sys 0.001s
I20260808 04:08:52.138796 1236 raft_consensus.cc:348] T 92d6d4818e6a4b6587573bb75bca5dbf P 078aaad2a15b48f5bc0917eff7c92fb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078aaad2a15b48f5bc0917eff7c92fb5" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 39335 } } peers { permanent_uuid: "190405558faf4fd292c52e057b794b33" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 38091 } } peers { permanent_uuid: "55739b89d3e64a528951c00448bc03bc" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 36851 } }
I20260808 04:08:52.139221 1236 raft_consensus.cc:374] T 92d6d4818e6a4b6587573bb75bca5dbf P 078aaad2a15b48f5bc0917eff7c92fb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:08:52.139334 1236 raft_consensus.cc:729] T 92d6d4818e6a4b6587573bb75bca5dbf P 078aaad2a15b48f5bc0917eff7c92fb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 078aaad2a15b48f5bc0917eff7c92fb5, State: Initialized, Role: FOLLOWER
I20260808 04:08:52.139308 1237 raft_consensus.cc:348] T 92d6d4818e6a4b6587573bb75bca5dbf P 55739b89d3e64a528951c00448bc03bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078aaad2a15b48f5bc0917eff7c92fb5" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 39335 } } peers { permanent_uuid: "190405558faf4fd292c52e057b794b33" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 38091 } } peers { permanent_uuid: "55739b89d3e64a528951c00448bc03bc" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 36851 } }
I20260808 04:08:52.139890 1237 raft_consensus.cc:374] T 92d6d4818e6a4b6587573bb75bca5dbf P 55739b89d3e64a528951c00448bc03bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:08:52.140023 1237 raft_consensus.cc:729] T 92d6d4818e6a4b6587573bb75bca5dbf P 55739b89d3e64a528951c00448bc03bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55739b89d3e64a528951c00448bc03bc, State: Initialized, Role: FOLLOWER
I20260808 04:08:52.139955 1236 consensus_queue.cc:260] T 92d6d4818e6a4b6587573bb75bca5dbf P 078aaad2a15b48f5bc0917eff7c92fb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078aaad2a15b48f5bc0917eff7c92fb5" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 39335 } } peers { permanent_uuid: "190405558faf4fd292c52e057b794b33" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 38091 } } peers { permanent_uuid: "55739b89d3e64a528951c00448bc03bc" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 36851 } }
I20260808 04:08:52.140691 1237 consensus_queue.cc:260] T 92d6d4818e6a4b6587573bb75bca5dbf P 55739b89d3e64a528951c00448bc03bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078aaad2a15b48f5bc0917eff7c92fb5" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 39335 } } peers { permanent_uuid: "190405558faf4fd292c52e057b794b33" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 38091 } } peers { permanent_uuid: "55739b89d3e64a528951c00448bc03bc" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 36851 } }
I20260808 04:08:52.141844 1036 heartbeater.cc:503] Master 127.5.83.62:38461 was elected leader, sending a full tablet report...
I20260808 04:08:52.142513 1236 ts_tablet_manager.cc:1428] T 92d6d4818e6a4b6587573bb75bca5dbf P 078aaad2a15b48f5bc0917eff7c92fb5: Time spent starting tablet: real 0.013s user 0.006s sys 0.007s
I20260808 04:08:52.143394 836 heartbeater.cc:503] Master 127.5.83.62:38461 was elected leader, sending a full tablet report...
I20260808 04:08:52.144188 1237 ts_tablet_manager.cc:1428] T 92d6d4818e6a4b6587573bb75bca5dbf P 55739b89d3e64a528951c00448bc03bc: Time spent starting tablet: real 0.015s user 0.010s sys 0.003s
W20260808 04:08:52.154568 837 tablet.cc:2369] T 92d6d4818e6a4b6587573bb75bca5dbf P 55739b89d3e64a528951c00448bc03bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:08:52.203806 1243 raft_consensus.cc:482] T 92d6d4818e6a4b6587573bb75bca5dbf P 190405558faf4fd292c52e057b794b33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:08:52.204106 1243 raft_consensus.cc:504] T 92d6d4818e6a4b6587573bb75bca5dbf P 190405558faf4fd292c52e057b794b33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078aaad2a15b48f5bc0917eff7c92fb5" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 39335 } } peers { permanent_uuid: "190405558faf4fd292c52e057b794b33" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 38091 } } peers { permanent_uuid: "55739b89d3e64a528951c00448bc03bc" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 36851 } }
I20260808 04:08:52.205890 1243 leader_election.cc:288] T 92d6d4818e6a4b6587573bb75bca5dbf P 190405558faf4fd292c52e057b794b33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 078aaad2a15b48f5bc0917eff7c92fb5 (127.5.83.2:39335), 55739b89d3e64a528951c00448bc03bc (127.5.83.1:36851)
I20260808 04:08:52.211474 1228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:52.206302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.3:51799 (local address 127.5.83.2:39335)
0808 04:08:52.206535 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:52.206541 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:52.206666 (+ 125us) server_negotiation.cc:407] Connection header received
0808 04:08:52.206864 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:52.206869 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:52.206947 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:52.207044 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:08:52.207482 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:52.208236 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:52.209886 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:52.210654 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:52.210767 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:52.210867 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:52.211107 (+ 240us) server_negotiation.cc:299] Negotiation successful
0808 04:08:52.211232 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260808 04:08:52.211509 1211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:52.206261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.2:39335 (local address 127.5.83.3:51799)
0808 04:08:52.206576 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:52.206594 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 04:08:52.206725 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:08:52.207106 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:08:52.207111 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:08:52.207143 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:08:52.207332 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:52.207344 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:52.208378 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:52.208384 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:52.209724 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:52.209737 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:52.210803 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:52.210808 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:52.210963 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:52.210988 (+ 25us) client_negotiation.cc:769] Sending connection context
0808 04:08:52.211068 (+ 80us) client_negotiation.cc:240] Negotiation successful
0808 04:08:52.211203 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":1}
I20260808 04:08:52.212231 1248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:52.206399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:36851 (local address 127.5.83.3:40751)
0808 04:08:52.207106 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:52.207122 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:08:52.207234 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:08:52.207556 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:08:52.207561 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:08:52.207586 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:08:52.207915 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:52.207926 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:52.209083 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:52.209089 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:52.210467 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:52.210480 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:52.211580 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:52.211585 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:52.211764 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:52.211794 (+ 30us) client_negotiation.cc:769] Sending connection context
0808 04:08:52.211892 (+ 98us) client_negotiation.cc:240] Negotiation successful
0808 04:08:52.211978 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"mutex_wait_us":40,"thread_start_us":370,"threads_started":1}
I20260808 04:08:52.212313 1234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:52.206354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.3:40751 (local address 127.5.83.1:36851)
0808 04:08:52.206532 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:52.206538 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:52.207202 (+ 664us) server_negotiation.cc:407] Connection header received
0808 04:08:52.207345 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:52.207349 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:52.207440 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:52.207518 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:08:52.208068 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:52.208913 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:52.210655 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:52.211411 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:52.211527 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:52.211612 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:52.211944 (+ 332us) server_negotiation.cc:299] Negotiation successful
0808 04:08:52.212071 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 04:08:52.212730 970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d6d4818e6a4b6587573bb75bca5dbf" candidate_uuid: "190405558faf4fd292c52e057b794b33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "078aaad2a15b48f5bc0917eff7c92fb5" is_pre_election: true
I20260808 04:08:52.213199 970 raft_consensus.cc:2454] T 92d6d4818e6a4b6587573bb75bca5dbf P 078aaad2a15b48f5bc0917eff7c92fb5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 190405558faf4fd292c52e057b794b33 in term 0.
I20260808 04:08:52.213122 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d6d4818e6a4b6587573bb75bca5dbf" candidate_uuid: "190405558faf4fd292c52e057b794b33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55739b89d3e64a528951c00448bc03bc" is_pre_election: true
I20260808 04:08:52.213567 772 raft_consensus.cc:2454] T 92d6d4818e6a4b6587573bb75bca5dbf P 55739b89d3e64a528951c00448bc03bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 190405558faf4fd292c52e057b794b33 in term 0.
I20260808 04:08:52.214001 1062 leader_election.cc:302] T 92d6d4818e6a4b6587573bb75bca5dbf P 190405558faf4fd292c52e057b794b33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 078aaad2a15b48f5bc0917eff7c92fb5, 190405558faf4fd292c52e057b794b33; no voters:
I20260808 04:08:52.214412 1243 raft_consensus.cc:2781] T 92d6d4818e6a4b6587573bb75bca5dbf P 190405558faf4fd292c52e057b794b33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:08:52.214545 1243 raft_consensus.cc:482] T 92d6d4818e6a4b6587573bb75bca5dbf P 190405558faf4fd292c52e057b794b33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:08:52.214648 1243 raft_consensus.cc:3037] T 92d6d4818e6a4b6587573bb75bca5dbf P 190405558faf4fd292c52e057b794b33 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:08:52.216936 1243 raft_consensus.cc:504] T 92d6d4818e6a4b6587573bb75bca5dbf P 190405558faf4fd292c52e057b794b33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078aaad2a15b48f5bc0917eff7c92fb5" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 39335 } } peers { permanent_uuid: "190405558faf4fd292c52e057b794b33" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 38091 } } peers { permanent_uuid: "55739b89d3e64a528951c00448bc03bc" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 36851 } }
I20260808 04:08:52.217475 1243 leader_election.cc:288] T 92d6d4818e6a4b6587573bb75bca5dbf P 190405558faf4fd292c52e057b794b33 [CANDIDATE]: Term 1 election: Requested vote from peers 078aaad2a15b48f5bc0917eff7c92fb5 (127.5.83.2:39335), 55739b89d3e64a528951c00448bc03bc (127.5.83.1:36851)
I20260808 04:08:52.217839 970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d6d4818e6a4b6587573bb75bca5dbf" candidate_uuid: "190405558faf4fd292c52e057b794b33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "078aaad2a15b48f5bc0917eff7c92fb5"
I20260808 04:08:52.217969 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d6d4818e6a4b6587573bb75bca5dbf" candidate_uuid: "190405558faf4fd292c52e057b794b33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55739b89d3e64a528951c00448bc03bc"
I20260808 04:08:52.218052 970 raft_consensus.cc:3037] T 92d6d4818e6a4b6587573bb75bca5dbf P 078aaad2a15b48f5bc0917eff7c92fb5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:08:52.218178 772 raft_consensus.cc:3037] T 92d6d4818e6a4b6587573bb75bca5dbf P 55739b89d3e64a528951c00448bc03bc [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:08:52.221298 772 raft_consensus.cc:2454] T 92d6d4818e6a4b6587573bb75bca5dbf P 55739b89d3e64a528951c00448bc03bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 190405558faf4fd292c52e057b794b33 in term 1.
I20260808 04:08:52.221298 970 raft_consensus.cc:2454] T 92d6d4818e6a4b6587573bb75bca5dbf P 078aaad2a15b48f5bc0917eff7c92fb5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 190405558faf4fd292c52e057b794b33 in term 1.
I20260808 04:08:52.221657 1062 leader_election.cc:302] T 92d6d4818e6a4b6587573bb75bca5dbf P 190405558faf4fd292c52e057b794b33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 078aaad2a15b48f5bc0917eff7c92fb5, 190405558faf4fd292c52e057b794b33; no voters:
I20260808 04:08:52.221930 1243 raft_consensus.cc:2781] T 92d6d4818e6a4b6587573bb75bca5dbf P 190405558faf4fd292c52e057b794b33 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:08:52.222420 1243 raft_consensus.cc:686] T 92d6d4818e6a4b6587573bb75bca5dbf P 190405558faf4fd292c52e057b794b33 [term 1 LEADER]: Becoming Leader. State: Replica: 190405558faf4fd292c52e057b794b33, State: Running, Role: LEADER
I20260808 04:08:52.222945 1243 consensus_queue.cc:237] T 92d6d4818e6a4b6587573bb75bca5dbf P 190405558faf4fd292c52e057b794b33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078aaad2a15b48f5bc0917eff7c92fb5" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 39335 } } peers { permanent_uuid: "190405558faf4fd292c52e057b794b33" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 38091 } } peers { permanent_uuid: "55739b89d3e64a528951c00448bc03bc" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 36851 } }
I20260808 04:08:52.227662 575 catalog_manager.cc:5495] T 92d6d4818e6a4b6587573bb75bca5dbf P 190405558faf4fd292c52e057b794b33 reported cstate change: term changed from 0 to 1, leader changed from <none> to 190405558faf4fd292c52e057b794b33 (127.5.83.3). New cstate: current_term: 1 leader_uuid: "190405558faf4fd292c52e057b794b33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078aaad2a15b48f5bc0917eff7c92fb5" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 39335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "190405558faf4fd292c52e057b794b33" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 38091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "55739b89d3e64a528951c00448bc03bc" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 36851 } health_report { overall_health: UNKNOWN } } }
I20260808 04:08:52.256880 1232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:52.251741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38716 (local address 127.5.83.3:38091)
0808 04:08:52.251966 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:52.251972 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:52.252042 (+ 70us) server_negotiation.cc:407] Connection header received
0808 04:08:52.252205 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:52.252209 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:52.252293 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:52.252413 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:08:52.252767 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:52.253507 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:52.254941 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:52.255338 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:52.255427 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:52.255927 (+ 500us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:08:52.256385 (+ 458us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:08:52.256513 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:52.256596 (+ 83us) server_negotiation.cc:299] Negotiation successful
0808 04:08:52.256630 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260808 04:08:52.264479 772 raft_consensus.cc:1261] T 92d6d4818e6a4b6587573bb75bca5dbf P 55739b89d3e64a528951c00448bc03bc [term 1 FOLLOWER]: Refusing update from remote peer 190405558faf4fd292c52e057b794b33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:08:52.264516 970 raft_consensus.cc:1261] T 92d6d4818e6a4b6587573bb75bca5dbf P 078aaad2a15b48f5bc0917eff7c92fb5 [term 1 FOLLOWER]: Refusing update from remote peer 190405558faf4fd292c52e057b794b33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:08:52.265363 1249 consensus_queue.cc:1035] T 92d6d4818e6a4b6587573bb75bca5dbf P 190405558faf4fd292c52e057b794b33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55739b89d3e64a528951c00448bc03bc" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 36851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:08:52.265681 1243 consensus_queue.cc:1035] T 92d6d4818e6a4b6587573bb75bca5dbf P 190405558faf4fd292c52e057b794b33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078aaad2a15b48f5bc0917eff7c92fb5" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 39335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:08:52.275972 1254 mvcc.cc:205] Tried to move back new op lower bound from 7316120093745930240 to 7316120093588987904. Current Snapshot: MvccSnapshot[applied={T|T < 7316120093745930240}]
I20260808 04:08:52.290891 575 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44604:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:08:52.291169 575 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44604:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:08:52.294661 575 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a3f52bb84d5241d89f64ac6e9bfd1b65: Sending DeleteTablet for 3 replicas of tablet 92d6d4818e6a4b6587573bb75bca5dbf
I20260808 04:08:52.295651 940 tablet_service.cc:1514] Processing DeleteTablet for tablet 92d6d4818e6a4b6587573bb75bca5dbf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:08:52 UTC) from {username='slave'} at 127.0.0.1:49808
I20260808 04:08:52.295827 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 92d6d4818e6a4b6587573bb75bca5dbf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:08:52 UTC) from {username='slave'} at 127.0.0.1:59148
I20260808 04:08:52.296034 1129 tablet_service.cc:1514] Processing DeleteTablet for tablet 92d6d4818e6a4b6587573bb75bca5dbf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:08:52 UTC) from {username='slave'} at 127.0.0.1:38710
I20260808 04:08:52.297405 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 658
I20260808 04:08:52.328959 1272 tablet_replica.cc:330] T 92d6d4818e6a4b6587573bb75bca5dbf P 55739b89d3e64a528951c00448bc03bc: stopping tablet replica
I20260808 04:08:52.329478 1272 raft_consensus.cc:2229] T 92d6d4818e6a4b6587573bb75bca5dbf P 55739b89d3e64a528951c00448bc03bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:08:52.330039 1272 raft_consensus.cc:2258] T 92d6d4818e6a4b6587573bb75bca5dbf P 55739b89d3e64a528951c00448bc03bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:08:52.333045 1271 tablet_replica.cc:330] T 92d6d4818e6a4b6587573bb75bca5dbf P 078aaad2a15b48f5bc0917eff7c92fb5: stopping tablet replica
I20260808 04:08:52.333504 1271 raft_consensus.cc:2229] T 92d6d4818e6a4b6587573bb75bca5dbf P 078aaad2a15b48f5bc0917eff7c92fb5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:08:52.333997 1271 raft_consensus.cc:2258] T 92d6d4818e6a4b6587573bb75bca5dbf P 078aaad2a15b48f5bc0917eff7c92fb5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:08:52.335829 1272 ts_tablet_manager.cc:1905] T 92d6d4818e6a4b6587573bb75bca5dbf P 55739b89d3e64a528951c00448bc03bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:08:52.337826 1271 ts_tablet_manager.cc:1905] T 92d6d4818e6a4b6587573bb75bca5dbf P 078aaad2a15b48f5bc0917eff7c92fb5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:08:52.337960 1234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:52.298813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59158 (local address 127.5.83.1:36851)
0808 04:08:52.299001 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:52.299007 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:52.323851 (+ 24844us) server_negotiation.cc:407] Connection header received
0808 04:08:52.324332 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:52.324339 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:52.324497 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:52.324577 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:08:52.325241 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:52.326475 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:52.330463 (+ 3988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:52.331141 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:52.331239 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:52.336218 (+ 4979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:08:52.336228 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:08:52.336244 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:08:52.337026 (+ 782us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:08:52.337089 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:08:52.337116 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:08:52.337227 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:52.337461 (+ 234us) server_negotiation.cc:299] Negotiation successful
0808 04:08:52.337586 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 04:08:52.339156 825 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:08:52.348047 1271 ts_tablet_manager.cc:1918] T 92d6d4818e6a4b6587573bb75bca5dbf P 078aaad2a15b48f5bc0917eff7c92fb5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:08:52.348752 1273 tablet_replica.cc:330] T 92d6d4818e6a4b6587573bb75bca5dbf P 190405558faf4fd292c52e057b794b33: stopping tablet replica
I20260808 04:08:52.349231 1273 raft_consensus.cc:2229] T 92d6d4818e6a4b6587573bb75bca5dbf P 190405558faf4fd292c52e057b794b33 [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:08:52.349942 1273 raft_consensus.cc:2258] T 92d6d4818e6a4b6587573bb75bca5dbf P 190405558faf4fd292c52e057b794b33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:08:52.350405 1271 log.cc:1196] T 92d6d4818e6a4b6587573bb75bca5dbf P 078aaad2a15b48f5bc0917eff7c92fb5: Deleting WAL directory at /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161826605655-5452-0/minicluster-data/ts-1/wal/wals/92d6d4818e6a4b6587573bb75bca5dbf
I20260808 04:08:52.352005 1273 ts_tablet_manager.cc:1905] T 92d6d4818e6a4b6587573bb75bca5dbf P 190405558faf4fd292c52e057b794b33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:08:52.352393 1271 ts_tablet_manager.cc:1939] T 92d6d4818e6a4b6587573bb75bca5dbf P 078aaad2a15b48f5bc0917eff7c92fb5: Deleting consensus metadata
I20260808 04:08:52.354494 554 catalog_manager.cc:4841] TS 078aaad2a15b48f5bc0917eff7c92fb5 (127.5.83.2:39335): tablet 92d6d4818e6a4b6587573bb75bca5dbf (table test-table [id=dd581f527bac45b29ed6cad205661c49]) successfully deleted
I20260808 04:08:52.360139 1273 ts_tablet_manager.cc:1918] T 92d6d4818e6a4b6587573bb75bca5dbf P 190405558faf4fd292c52e057b794b33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:08:52.360309 1273 log.cc:1196] T 92d6d4818e6a4b6587573bb75bca5dbf P 190405558faf4fd292c52e057b794b33: Deleting WAL directory at /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161826605655-5452-0/minicluster-data/ts-2/wal/wals/92d6d4818e6a4b6587573bb75bca5dbf
I20260808 04:08:52.360844 1273 ts_tablet_manager.cc:1939] T 92d6d4818e6a4b6587573bb75bca5dbf P 190405558faf4fd292c52e057b794b33: Deleting consensus metadata
I20260808 04:08:52.362601 555 catalog_manager.cc:4841] TS 190405558faf4fd292c52e057b794b33 (127.5.83.3:38091): tablet 92d6d4818e6a4b6587573bb75bca5dbf (table test-table [id=dd581f527bac45b29ed6cad205661c49]) successfully deleted
I20260808 04:08:52.429251 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 658
W20260808 04:08:52.451009 555 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.83.1:36851 (error 108)
I20260808 04:08:52.451472 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 843
I20260808 04:08:52.453265 1233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:52.452793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:36851 (local address 127.0.0.1:59166)
0808 04:08:52.452984 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:52.453095 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.1:36851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260808 04:08:52.453675 555 catalog_manager.cc:4584] TS 55739b89d3e64a528951c00448bc03bc (127.5.83.1:36851): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 92d6d4818e6a4b6587573bb75bca5dbf: Network error: Client connection negotiation failed: client connection to 127.5.83.1:36851: connect: Connection refused (error 111)
I20260808 04:08:52.459437 1228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:52.452474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49824 (local address 127.5.83.2:39335)
0808 04:08:52.452656 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:52.452662 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:52.452683 (+ 21us) server_negotiation.cc:407] Connection header received
0808 04:08:52.452869 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:52.452874 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:52.453010 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:52.453122 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:08:52.453592 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:52.454700 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:52.456147 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:52.456789 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:52.456928 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:52.457833 (+ 905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:08:52.457844 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:08:52.457861 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:08:52.458639 (+ 778us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:08:52.458697 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:08:52.458722 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:08:52.458830 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:52.458982 (+ 152us) server_negotiation.cc:299] Negotiation successful
0808 04:08:52.459091 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 04:08:52.460222 1031 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:08:52.507457 1233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:52.506897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:36851 (local address 127.0.0.1:59182)
0808 04:08:52.507142 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:52.507253 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.1:36851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260808 04:08:52.508636 1233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:52.508240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:36851 (local address 127.0.0.1:59194)
0808 04:08:52.508398 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:52.508485 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.1:36851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260808 04:08:52.546922 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 843
I20260808 04:08:52.568616 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 1042
I20260808 04:08:52.569775 1233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:52.569297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:36851 (local address 127.0.0.1:59196)
0808 04:08:52.569546 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:52.569619 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.1:36851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150}
I20260808 04:08:52.570770 1233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:52.570365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:36851 (local address 127.0.0.1:59206)
0808 04:08:52.570531 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:52.570607 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.1:36851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260808 04:08:52.577252 1232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:52.569460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38718 (local address 127.5.83.3:38091)
0808 04:08:52.569651 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:52.569657 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:52.570450 (+ 793us) server_negotiation.cc:407] Connection header received
0808 04:08:52.570527 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:52.570531 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:52.570633 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:52.570740 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:08:52.571454 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:52.572313 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:52.574218 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:52.574657 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:52.574753 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:52.575910 (+ 1157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:08:52.575918 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:08:52.575933 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:08:52.576538 (+ 605us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:08:52.576586 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:08:52.576607 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:08:52.576687 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:52.576845 (+ 158us) server_negotiation.cc:299] Negotiation successful
0808 04:08:52.576929 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260808 04:08:52.578028 1203 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:08:52.651414 1233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:52.650936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:36851 (local address 127.0.0.1:59218)
0808 04:08:52.651157 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:52.651248 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.1:36851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260808 04:08:52.652477 1233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:52.652097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:36851 (local address 127.0.0.1:59228)
0808 04:08:52.652291 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:52.652362 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.1:36851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260808 04:08:52.695463 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 1042
I20260808 04:08:52.719034 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 532
I20260808 04:08:52.720163 618 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:08:52.825800 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 532
2026-08-08T04:08:52Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2695 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:08:52 dist-test-slave-1l3l krb5kdc[1285](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:08:52 dist-test-slave-1l3l krb5kdc[1285](info): set up 2 sockets
Aug 08 04:08:52 dist-test-slave-1l3l krb5kdc[1285](info): commencing operation
krb5kdc: starting...
W20260808 04:08:54.024273 5452 mini_kdc.cc:121] Time spent starting KDC: real 1.156s user 0.006s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/krb5kdc/test-user.keytab.
Aug 08 04:08:54 dist-test-slave-1l3l krb5kdc[1285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162134, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:08:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:08:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.83.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab.
Entry for principal kudu/127.5.83.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab.
I20260808 04:08:54.283214 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:45587
--webserver_interface=127.5.83.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab
--principal=kudu/127.5.83.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:38653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.62:45587
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:08:54.602947 1300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:08:54.603209 1300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:08:54.603296 1300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:08:54.610342 1300 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:08:54.610446 1300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:08:54.610498 1300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:08:54.610553 1300 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:08:54.610597 1300 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:08:54.620021 1300 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:38653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.62:45587
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab
--principal=kudu/127.5.83.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:45587
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:08:54.621243 1300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:08:54.622468 1300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:08:54.633503 1307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:08:54.633473 1310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:08:54.633548 1312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:08:54.634581 1300 server_base.cc:1034] running on GCE node
I20260808 04:08:54.635365 1300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:08:54.636714 1300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:08:54.637916 1300 hybrid_clock.cc:648] HybridClock initialized: now 1786162134637866 us; error 41 us; skew 500 ppm
I20260808 04:08:54.638222 1300 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:08:54 dist-test-slave-1l3l krb5kdc[1285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162134, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:08:54.644038 1300 init.cc:377] Logged in from keytab as kudu/127.5.83.62@KRBTEST.COM (short username kudu)
I20260808 04:08:54.645422 1300 webserver.cc:460] Webserver started at http://127.5.83.62:40169/ using document root <none> and password file <none>
I20260808 04:08:54.646055 1300 fs_manager.cc:359] Metadata directory not provided
I20260808 04:08:54.646170 1300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:08:54.646454 1300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:08:54.649346 1300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/minicluster-data/master-0/data/instance:
uuid: "9504ad9897c841e8aa2f0a5442f1de68"
format_stamp: "Formatted at 2026-08-08 04:08:54 on dist-test-slave-1l3l"
server_key: "78a12b8ad3cb41cbf9876a8a0535d22a"
server_key_iv: "652610a32ba4c4b3f312c16c387fab46"
server_key_version: "encryptionkey@0"
I20260808 04:08:54.650071 1300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/minicluster-data/master-0/wal/instance:
uuid: "9504ad9897c841e8aa2f0a5442f1de68"
format_stamp: "Formatted at 2026-08-08 04:08:54 on dist-test-slave-1l3l"
server_key: "78a12b8ad3cb41cbf9876a8a0535d22a"
server_key_iv: "652610a32ba4c4b3f312c16c387fab46"
server_key_version: "encryptionkey@0"
I20260808 04:08:54.654919 1300 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 04:08:54.658236 1322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:08:54.659478 1300 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260808 04:08:54.659639 1300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/minicluster-data/master-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/minicluster-data/master-0/wal
uuid: "9504ad9897c841e8aa2f0a5442f1de68"
format_stamp: "Formatted at 2026-08-08 04:08:54 on dist-test-slave-1l3l"
server_key: "78a12b8ad3cb41cbf9876a8a0535d22a"
server_key_iv: "652610a32ba4c4b3f312c16c387fab46"
server_key_version: "encryptionkey@0"
I20260808 04:08:54.659863 1300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:08:54.677287 1300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:08:54.681123 1300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:08:54.681371 1300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:08:54.698203 1300 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.62:45587
I20260808 04:08:54.698237 1392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.62:45587 every 8 connection(s)
I20260808 04:08:54.699470 1300 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
I20260808 04:08:54.703055 1394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:08:54.706718 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 1300
I20260808 04:08:54.706899 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/minicluster-data/master-0/wal/instance
I20260808 04:08:54.707315 5452 external_mini_cluster.cc:1428] Setting key 528b01a0f9e16be1d3ad40a02f1ff800
I20260808 04:08:54.713595 1394 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9504ad9897c841e8aa2f0a5442f1de68: Bootstrap starting.
I20260808 04:08:54.716449 1394 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9504ad9897c841e8aa2f0a5442f1de68: Neither blocks nor log segments found. Creating new log.
I20260808 04:08:54.717357 1394 log.cc:824] T 00000000000000000000000000000000 P 9504ad9897c841e8aa2f0a5442f1de68: Log is configured to *not* fsync() on all Append() calls
I20260808 04:08:54.720489 1394 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9504ad9897c841e8aa2f0a5442f1de68: No bootstrap required, opened a new log
Aug 08 04:08:54 dist-test-slave-1l3l krb5kdc[1285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162134, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:08:54.728828 1394 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9504ad9897c841e8aa2f0a5442f1de68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9504ad9897c841e8aa2f0a5442f1de68" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 45587 } }
I20260808 04:08:54.729288 1394 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9504ad9897c841e8aa2f0a5442f1de68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:08:54.729416 1394 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9504ad9897c841e8aa2f0a5442f1de68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9504ad9897c841e8aa2f0a5442f1de68, State: Initialized, Role: FOLLOWER
I20260808 04:08:54.729947 1394 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9504ad9897c841e8aa2f0a5442f1de68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9504ad9897c841e8aa2f0a5442f1de68" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 45587 } }
I20260808 04:08:54.730186 1394 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9504ad9897c841e8aa2f0a5442f1de68 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:08:54.730320 1394 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9504ad9897c841e8aa2f0a5442f1de68 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:08:54.730468 1400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:54.709085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33538 (local address 127.5.83.62:45587)
0808 04:08:54.709813 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:54.709824 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:54.709863 (+ 39us) server_negotiation.cc:407] Connection header received
0808 04:08:54.712468 (+ 2605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:54.712490 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:54.712816 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:54.713140 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:08:54.714653 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:54.715988 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:54.717332 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:54.718236 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:54.718434 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:54.723141 (+ 4707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:08:54.723151 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:08:54.723163 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:08:54.723205 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:08:54.726182 (+ 2977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:08:54.726982 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:08:54.726988 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:08:54.726994 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:08:54.727158 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:08:54.727544 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:08:54.727547 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:08:54.727549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:08:54.728217 (+ 668us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:08:54.728429 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:54.728870 (+ 441us) server_negotiation.cc:299] Negotiation successful
0808 04:08:54.729269 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":341,"threads_started":1}
I20260808 04:08:54.730541 1394 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9504ad9897c841e8aa2f0a5442f1de68 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:08:54.732856 1394 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9504ad9897c841e8aa2f0a5442f1de68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9504ad9897c841e8aa2f0a5442f1de68" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 45587 } }
I20260808 04:08:54.733361 1394 leader_election.cc:302] T 00000000000000000000000000000000 P 9504ad9897c841e8aa2f0a5442f1de68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9504ad9897c841e8aa2f0a5442f1de68; no voters:
I20260808 04:08:54.733901 1394 leader_election.cc:288] T 00000000000000000000000000000000 P 9504ad9897c841e8aa2f0a5442f1de68 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:08:54.734021 1402 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9504ad9897c841e8aa2f0a5442f1de68 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:08:54.734606 1402 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9504ad9897c841e8aa2f0a5442f1de68 [term 1 LEADER]: Becoming Leader. State: Replica: 9504ad9897c841e8aa2f0a5442f1de68, State: Running, Role: LEADER
I20260808 04:08:54.735086 1402 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9504ad9897c841e8aa2f0a5442f1de68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9504ad9897c841e8aa2f0a5442f1de68" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 45587 } }
I20260808 04:08:54.735733 1394 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9504ad9897c841e8aa2f0a5442f1de68 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:08:54.737977 1404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9504ad9897c841e8aa2f0a5442f1de68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9504ad9897c841e8aa2f0a5442f1de68. Latest consensus state: current_term: 1 leader_uuid: "9504ad9897c841e8aa2f0a5442f1de68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9504ad9897c841e8aa2f0a5442f1de68" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 45587 } } }
I20260808 04:08:54.738950 1404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9504ad9897c841e8aa2f0a5442f1de68 [sys.catalog]: This master's current role is: LEADER
I20260808 04:08:54.738606 1403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9504ad9897c841e8aa2f0a5442f1de68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9504ad9897c841e8aa2f0a5442f1de68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9504ad9897c841e8aa2f0a5442f1de68" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 45587 } } }
I20260808 04:08:54.739426 1403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9504ad9897c841e8aa2f0a5442f1de68 [sys.catalog]: This master's current role is: LEADER
I20260808 04:08:54.742213 1421 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:08:54.746542 1421 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:08:54.754112 1421 catalog_manager.cc:1347] Generated new cluster ID: 420cde7cc7414323a4c6c2db3c46d104
I20260808 04:08:54.754205 1421 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:08:54.780495 1421 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:08:54.782353 1421 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:08:54.794956 1421 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9504ad9897c841e8aa2f0a5442f1de68: Generated new TSK 0
I20260808 04:08:54.796010 1421 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.83.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.1.keytab.
Entry for principal kudu/127.5.83.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.1.keytab.
I20260808 04:08:54.865093 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.1:0
--local_ip_for_outbound_sockets=127.5.83.1
--webserver_interface=127.5.83.1
--webserver_port=0
--tserver_master_addrs=127.5.83.62:45587
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.1.keytab
--principal=kudu/127.5.83.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:38653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:08:55.195118 1442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:08:55.195403 1442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:08:55.195537 1442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:08:55.202531 1442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:08:55.202627 1442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:08:55.202776 1442 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.1
I20260808 04:08:55.212090 1442 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:38653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.1.keytab
--principal=kudu/127.5.83.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.83.1
--webserver_port=0
--tserver_master_addrs=127.5.83.62:45587
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.1
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:08:55.213274 1442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:08:55.214547 1442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:08:55.227371 1447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:08:55.227284 1451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:08:55.227332 1448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:08:55.228474 1442 server_base.cc:1034] running on GCE node
I20260808 04:08:55.229059 1442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:08:55.231551 1442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:08:55.232797 1442 hybrid_clock.cc:648] HybridClock initialized: now 1786162135232724 us; error 45 us; skew 500 ppm
I20260808 04:08:55.233188 1442 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:08:55 dist-test-slave-1l3l krb5kdc[1285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162135, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:08:55.236997 1442 init.cc:377] Logged in from keytab as kudu/127.5.83.1@KRBTEST.COM (short username kudu)
I20260808 04:08:55.238348 1442 webserver.cc:460] Webserver started at http://127.5.83.1:38125/ using document root <none> and password file <none>
I20260808 04:08:55.238970 1442 fs_manager.cc:359] Metadata directory not provided
I20260808 04:08:55.239068 1442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:08:55.239351 1442 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:08:55.242228 1442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/minicluster-data/ts-0/data/instance:
uuid: "0a9d391ff4724d4aa535d40a05a03a34"
format_stamp: "Formatted at 2026-08-08 04:08:55 on dist-test-slave-1l3l"
server_key: "6b0ce82226f0784af5113a6f127d5645"
server_key_iv: "8082f4ede37245716cfaf38dcdb58a2f"
server_key_version: "encryptionkey@0"
I20260808 04:08:55.242985 1442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/minicluster-data/ts-0/wal/instance:
uuid: "0a9d391ff4724d4aa535d40a05a03a34"
format_stamp: "Formatted at 2026-08-08 04:08:55 on dist-test-slave-1l3l"
server_key: "6b0ce82226f0784af5113a6f127d5645"
server_key_iv: "8082f4ede37245716cfaf38dcdb58a2f"
server_key_version: "encryptionkey@0"
I20260808 04:08:55.247813 1442 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.001s
I20260808 04:08:55.250945 1464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:08:55.252200 1442 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:08:55.252331 1442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/minicluster-data/ts-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/minicluster-data/ts-0/wal
uuid: "0a9d391ff4724d4aa535d40a05a03a34"
format_stamp: "Formatted at 2026-08-08 04:08:55 on dist-test-slave-1l3l"
server_key: "6b0ce82226f0784af5113a6f127d5645"
server_key_iv: "8082f4ede37245716cfaf38dcdb58a2f"
server_key_version: "encryptionkey@0"
I20260808 04:08:55.252486 1442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:08:55.271051 1442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:08:55.274662 1442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:08:55.274886 1442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:08:55.275780 1442 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:08:55.277004 1442 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:08:55.277074 1442 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:08:55.277164 1442 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:08:55.277207 1442 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:08:55.307257 1442 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.1:39401
I20260808 04:08:55.307300 1635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.1:39401 every 8 connection(s)
I20260808 04:08:55.308516 1442 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
I20260808 04:08:55.313879 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 1442
I20260808 04:08:55.314083 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/minicluster-data/ts-0/wal/instance
I20260808 04:08:55.314646 5452 external_mini_cluster.cc:1428] Setting key 4126c2080cda5260df3b104538577c6f
Aug 08 04:08:55 dist-test-slave-1l3l krb5kdc[1285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162135, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.1@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:08:55.328624 1641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:55.310797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.1:35439 (local address 127.5.83.62:45587)
0808 04:08:55.311183 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:55.311188 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:55.312066 (+ 878us) server_negotiation.cc:407] Connection header received
0808 04:08:55.313969 (+ 1903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:55.313975 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:55.314081 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:55.314196 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:08:55.316072 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:55.316999 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:55.319092 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:55.319490 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:55.319584 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:55.323331 (+ 3747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:08:55.323336 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:08:55.323339 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:08:55.323371 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:08:55.325826 (+ 2455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:08:55.326567 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:08:55.326571 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:08:55.326573 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:08:55.326703 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:08:55.327100 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:08:55.327103 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:08:55.327105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:08:55.327350 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:08:55.327461 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:55.328209 (+ 748us) server_negotiation.cc:299] Negotiation successful
0808 04:08:55.328372 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":202,"threads_started":1}
I20260808 04:08:55.329775 1642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:55.311215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:45587 (local address 127.5.83.1:35439)
0808 04:08:55.311917 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:55.311952 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0808 04:08:55.313654 (+ 1702us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:08:55.314487 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:08:55.314498 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:08:55.315212 (+ 714us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:08:55.315913 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:55.315934 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:55.317159 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:55.317165 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:55.318954 (+ 1789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:55.318969 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:55.319730 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:55.319737 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:55.319987 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:55.320647 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:08:55.320679 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:08:55.323119 (+ 2440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:08:55.325954 (+ 2835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:08:55.325961 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:08:55.325975 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:08:55.326471 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:08:55.326792 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:08:55.326796 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:08:55.326798 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:08:55.327011 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:08:55.327464 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:08:55.327470 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:08:55.327849 (+ 379us) client_negotiation.cc:769] Sending connection context
0808 04:08:55.328122 (+ 273us) client_negotiation.cc:240] Negotiation successful
0808 04:08:55.328590 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":269,"threads_started":1}
I20260808 04:08:55.331171 1636 heartbeater.cc:347] Connected to a master server at 127.5.83.62:45587
I20260808 04:08:55.331471 1636 heartbeater.cc:464] Registering TS with master...
I20260808 04:08:55.332300 1636 heartbeater.cc:511] Master 127.5.83.62:45587 requested a full tablet report, sending...
I20260808 04:08:55.334363 1346 ts_manager.cc:194] Registered new tserver with Master: 0a9d391ff4724d4aa535d40a05a03a34 (127.5.83.1:39401)
I20260808 04:08:55.335852 1346 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.1@KRBTEST.COM'} at 127.5.83.1:35439
WARNING: no policy specified for kudu/127.5.83.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.2.keytab.
Entry for principal kudu/127.5.83.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.2.keytab.
I20260808 04:08:55.376966 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.2:0
--local_ip_for_outbound_sockets=127.5.83.2
--webserver_interface=127.5.83.2
--webserver_port=0
--tserver_master_addrs=127.5.83.62:45587
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.2.keytab
--principal=kudu/127.5.83.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:38653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:08:55.695433 1647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:08:55.695680 1647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:08:55.695832 1647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:08:55.702860 1647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:08:55.702971 1647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:08:55.703122 1647 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.2
I20260808 04:08:55.712404 1647 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:38653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.2.keytab
--principal=kudu/127.5.83.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.83.2
--webserver_port=0
--tserver_master_addrs=127.5.83.62:45587
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.2
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:08:55.713518 1647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:08:55.714823 1647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:08:55.727474 1658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:08:55.727520 1656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:08:55.727546 1655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:08:55.728211 1647 server_base.cc:1034] running on GCE node
I20260808 04:08:55.728755 1647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:08:55.729628 1647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:08:55.730811 1647 hybrid_clock.cc:648] HybridClock initialized: now 1786162135730768 us; error 34 us; skew 500 ppm
I20260808 04:08:55.731091 1647 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:08:55 dist-test-slave-1l3l krb5kdc[1285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162135, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:08:55.733963 1647 init.cc:377] Logged in from keytab as kudu/127.5.83.2@KRBTEST.COM (short username kudu)
I20260808 04:08:55.735236 1647 webserver.cc:460] Webserver started at http://127.5.83.2:37149/ using document root <none> and password file <none>
I20260808 04:08:55.735961 1647 fs_manager.cc:359] Metadata directory not provided
I20260808 04:08:55.736073 1647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:08:55.736372 1647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:08:55.739208 1647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/minicluster-data/ts-1/data/instance:
uuid: "0e416e8debae4680af6860cf9d0dce7f"
format_stamp: "Formatted at 2026-08-08 04:08:55 on dist-test-slave-1l3l"
server_key: "83705e4dd8c016fc373199e604d603d2"
server_key_iv: "9323889c0912378ebe466130e106a3b8"
server_key_version: "encryptionkey@0"
I20260808 04:08:55.740023 1647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/minicluster-data/ts-1/wal/instance:
uuid: "0e416e8debae4680af6860cf9d0dce7f"
format_stamp: "Formatted at 2026-08-08 04:08:55 on dist-test-slave-1l3l"
server_key: "83705e4dd8c016fc373199e604d603d2"
server_key_iv: "9323889c0912378ebe466130e106a3b8"
server_key_version: "encryptionkey@0"
I20260808 04:08:55.744726 1647 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260808 04:08:55.747823 1669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:08:55.749014 1647 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 04:08:55.749154 1647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/minicluster-data/ts-1/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/minicluster-data/ts-1/wal
uuid: "0e416e8debae4680af6860cf9d0dce7f"
format_stamp: "Formatted at 2026-08-08 04:08:55 on dist-test-slave-1l3l"
server_key: "83705e4dd8c016fc373199e604d603d2"
server_key_iv: "9323889c0912378ebe466130e106a3b8"
server_key_version: "encryptionkey@0"
I20260808 04:08:55.749327 1647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:08:55.774461 1647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:08:55.778047 1647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:08:55.778282 1647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:08:55.779129 1647 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:08:55.780431 1647 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:08:55.780503 1647 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:08:55.780587 1647 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:08:55.780633 1647 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:08:55.809734 1647 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.2:43895
I20260808 04:08:55.809799 1821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.2:43895 every 8 connection(s)
I20260808 04:08:55.811017 1647 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
I20260808 04:08:55.820307 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 1647
I20260808 04:08:55.820497 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/minicluster-data/ts-1/wal/instance
I20260808 04:08:55.820927 5452 external_mini_cluster.cc:1428] Setting key a95a7467f2ea3cd61d1bb3cc2efc29f8
Aug 08 04:08:55 dist-test-slave-1l3l krb5kdc[1285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162135, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.2@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:08:55.829586 1641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:55.813466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.2:43275 (local address 127.5.83.62:45587)
0808 04:08:55.813640 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:55.813644 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:55.814687 (+ 1043us) server_negotiation.cc:407] Connection header received
0808 04:08:55.816537 (+ 1850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:55.816543 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:55.816625 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:55.816715 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:08:55.818132 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:55.818833 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:55.820401 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:55.820882 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:55.821000 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:55.824309 (+ 3309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:08:55.824313 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:08:55.824317 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:08:55.824349 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:08:55.826731 (+ 2382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:08:55.827397 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:08:55.827401 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:08:55.827403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:08:55.827524 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:08:55.827971 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:08:55.827975 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:08:55.827977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:08:55.828232 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:08:55.828367 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:55.829013 (+ 646us) server_negotiation.cc:299] Negotiation successful
0808 04:08:55.829238 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 04:08:55.830570 1824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:55.813856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:45587 (local address 127.5.83.2:43275)
0808 04:08:55.814534 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:55.814577 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 04:08:55.816281 (+ 1704us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:08:55.816948 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:08:55.816956 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:08:55.817444 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:08:55.817972 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:55.817988 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:55.818987 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:55.818991 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:55.820251 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:55.820268 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:55.821108 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:55.821116 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:55.821315 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:55.821953 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:08:55.821974 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:08:55.824109 (+ 2135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:08:55.826858 (+ 2749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:08:55.826867 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:08:55.826889 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:08:55.827274 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:08:55.827626 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:08:55.827631 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:08:55.827635 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:08:55.827855 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:08:55.828362 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:08:55.828370 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:08:55.828671 (+ 301us) client_negotiation.cc:769] Sending connection context
0808 04:08:55.828901 (+ 230us) client_negotiation.cc:240] Negotiation successful
0808 04:08:55.829387 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":271,"threads_started":1}
I20260808 04:08:55.832183 1822 heartbeater.cc:347] Connected to a master server at 127.5.83.62:45587
I20260808 04:08:55.832484 1822 heartbeater.cc:464] Registering TS with master...
I20260808 04:08:55.833145 1822 heartbeater.cc:511] Master 127.5.83.62:45587 requested a full tablet report, sending...
I20260808 04:08:55.834537 1346 ts_manager.cc:194] Registered new tserver with Master: 0e416e8debae4680af6860cf9d0dce7f (127.5.83.2:43895)
I20260808 04:08:55.835368 1346 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.2@KRBTEST.COM'} at 127.5.83.2:43275
WARNING: no policy specified for kudu/127.5.83.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.3.keytab.
Entry for principal kudu/127.5.83.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.3.keytab.
I20260808 04:08:55.880967 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.3:0
--local_ip_for_outbound_sockets=127.5.83.3
--webserver_interface=127.5.83.3
--webserver_port=0
--tserver_master_addrs=127.5.83.62:45587
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.3.keytab
--principal=kudu/127.5.83.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:38653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:08:56.198141 1828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:08:56.198396 1828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:08:56.198508 1828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:08:56.205449 1828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:08:56.205551 1828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:08:56.205662 1828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.3
I20260808 04:08:56.214869 1828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:38653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.3.keytab
--principal=kudu/127.5.83.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.83.3
--webserver_port=0
--tserver_master_addrs=127.5.83.62:45587
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.3
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:08:56.216045 1828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:08:56.217238 1828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:08:56.229544 1836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:08:56.229648 1838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:08:56.229622 1835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:08:56.230029 1828 server_base.cc:1034] running on GCE node
I20260808 04:08:56.230708 1828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:08:56.231624 1828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:08:56.232877 1828 hybrid_clock.cc:648] HybridClock initialized: now 1786162136232815 us; error 55 us; skew 500 ppm
I20260808 04:08:56.233171 1828 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:08:56 dist-test-slave-1l3l krb5kdc[1285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162136, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:08:56.236433 1828 init.cc:377] Logged in from keytab as kudu/127.5.83.3@KRBTEST.COM (short username kudu)
I20260808 04:08:56.238624 1828 webserver.cc:460] Webserver started at http://127.5.83.3:33651/ using document root <none> and password file <none>
I20260808 04:08:56.239336 1828 fs_manager.cc:359] Metadata directory not provided
I20260808 04:08:56.239487 1828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:08:56.239835 1828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:08:56.242615 1828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/minicluster-data/ts-2/data/instance:
uuid: "da20ac16c210419bbc22b151f1e10239"
format_stamp: "Formatted at 2026-08-08 04:08:56 on dist-test-slave-1l3l"
server_key: "88461ad83a146a0019b03f03b81ba7c4"
server_key_iv: "2b328563d3dcaaa9e456accaa677bae1"
server_key_version: "encryptionkey@0"
I20260808 04:08:56.243408 1828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/minicluster-data/ts-2/wal/instance:
uuid: "da20ac16c210419bbc22b151f1e10239"
format_stamp: "Formatted at 2026-08-08 04:08:56 on dist-test-slave-1l3l"
server_key: "88461ad83a146a0019b03f03b81ba7c4"
server_key_iv: "2b328563d3dcaaa9e456accaa677bae1"
server_key_version: "encryptionkey@0"
I20260808 04:08:56.248307 1828 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260808 04:08:56.251448 1854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:08:56.252779 1828 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260808 04:08:56.252933 1828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/minicluster-data/ts-2/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/minicluster-data/ts-2/wal
uuid: "da20ac16c210419bbc22b151f1e10239"
format_stamp: "Formatted at 2026-08-08 04:08:56 on dist-test-slave-1l3l"
server_key: "88461ad83a146a0019b03f03b81ba7c4"
server_key_iv: "2b328563d3dcaaa9e456accaa677bae1"
server_key_version: "encryptionkey@0"
I20260808 04:08:56.253103 1828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:08:56.264403 1828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:08:56.268338 1828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:08:56.268563 1828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:08:56.269402 1828 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:08:56.270586 1828 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:08:56.270656 1828 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:08:56.270741 1828 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:08:56.270783 1828 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:08:56.300331 1828 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.3:41775
I20260808 04:08:56.300367 2022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.3:41775 every 8 connection(s)
I20260808 04:08:56.301498 1828 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
I20260808 04:08:56.304895 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 1828
I20260808 04:08:56.305089 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/minicluster-data/ts-2/wal/instance
I20260808 04:08:56.305739 5452 external_mini_cluster.cc:1428] Setting key a26c30f2103e402a339a152992318dee
Aug 08 04:08:56 dist-test-slave-1l3l krb5kdc[1285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162136, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.3@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:08:56.321904 1641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:56.303847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.3:54663 (local address 127.5.83.62:45587)
0808 04:08:56.304031 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:56.304036 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:56.305319 (+ 1283us) server_negotiation.cc:407] Connection header received
0808 04:08:56.307647 (+ 2328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:56.307653 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:56.307763 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:56.307860 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:08:56.309712 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:56.310502 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:56.312709 (+ 2207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:56.313104 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:56.313197 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:56.316349 (+ 3152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:08:56.316354 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:08:56.316357 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:08:56.316392 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:08:56.318927 (+ 2535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:08:56.319779 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:08:56.319785 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:08:56.319788 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:08:56.319968 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:08:56.320429 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:08:56.320432 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:08:56.320434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:08:56.320774 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:08:56.320910 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:56.321494 (+ 584us) server_negotiation.cc:299] Negotiation successful
0808 04:08:56.321655 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 04:08:56.322736 2027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:56.304284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:45587 (local address 127.5.83.3:54663)
0808 04:08:56.305145 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:56.305194 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0808 04:08:56.307355 (+ 2161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:08:56.308116 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:08:56.308127 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:08:56.308807 (+ 680us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:08:56.309544 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:56.309570 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:56.310647 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:56.310653 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:56.312576 (+ 1923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:56.312591 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:56.313280 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:56.313285 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:56.313473 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:56.314079 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:08:56.314101 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:08:56.316141 (+ 2040us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:08:56.319054 (+ 2913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:08:56.319060 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:08:56.319070 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:08:56.319612 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:08:56.320089 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:08:56.320092 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:08:56.320094 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:08:56.320315 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:08:56.320901 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:08:56.320908 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:08:56.321184 (+ 276us) client_negotiation.cc:769] Sending connection context
0808 04:08:56.321391 (+ 207us) client_negotiation.cc:240] Negotiation successful
0808 04:08:56.321733 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":327,"threads_started":1}
I20260808 04:08:56.323939 2023 heartbeater.cc:347] Connected to a master server at 127.5.83.62:45587
I20260808 04:08:56.324209 2023 heartbeater.cc:464] Registering TS with master...
I20260808 04:08:56.324846 2023 heartbeater.cc:511] Master 127.5.83.62:45587 requested a full tablet report, sending...
I20260808 04:08:56.326123 1346 ts_manager.cc:194] Registered new tserver with Master: da20ac16c210419bbc22b151f1e10239 (127.5.83.3:41775)
I20260808 04:08:56.326874 1346 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.3@KRBTEST.COM'} at 127.5.83.3:54663
I20260808 04:08:56.332755 5452 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:08:56.338969 1636 heartbeater.cc:503] Master 127.5.83.62:45587 was elected leader, sending a full tablet report...
I20260808 04:08:56.348003 1641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:56.336528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33554 (local address 127.5.83.62:45587)
0808 04:08:56.336662 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:56.336668 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:56.337089 (+ 421us) server_negotiation.cc:407] Connection header received
0808 04:08:56.337249 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:56.337253 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:56.337308 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:56.337382 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:08:56.338819 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:56.339622 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:56.340978 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:56.341374 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:56.341468 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:56.343589 (+ 2121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:08:56.343593 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:08:56.343597 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:08:56.343625 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:08:56.346017 (+ 2392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:08:56.346531 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:08:56.346535 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:08:56.346537 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:08:56.346633 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:08:56.346909 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:08:56.346912 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:08:56.346914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:08:56.347183 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:08:56.347353 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:56.347591 (+ 238us) server_negotiation.cc:299] Negotiation successful
0808 04:08:56.347771 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 04:08:56.351212 1346 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33554:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:08:56.358065 1346 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:08:56.400143 2046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:56.369810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48070 (local address 127.5.83.1:39401)
0808 04:08:56.388139 (+ 18329us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:56.388150 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:56.391568 (+ 3418us) server_negotiation.cc:407] Connection header received
0808 04:08:56.392060 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:56.392070 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:56.392340 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:56.392517 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:08:56.393261 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:56.394825 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:56.397259 (+ 2434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:56.398682 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:56.398806 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:56.399031 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:56.399619 (+ 588us) server_negotiation.cc:299] Negotiation successful
0808 04:08:56.399818 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18133,"thread_start_us":17953,"threads_started":1}
I20260808 04:08:56.400223 2042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:56.369236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:39401 (local address 127.0.0.1:48070)
0808 04:08:56.391443 (+ 22207us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:56.391461 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 04:08:56.391862 (+ 401us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:08:56.392721 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:08:56.392728 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:08:56.392758 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:08:56.393096 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:56.393109 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:56.395016 (+ 1907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:56.395021 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:56.397111 (+ 2090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:56.397128 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:56.399168 (+ 2040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:56.399174 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:56.399386 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:56.399450 (+ 64us) client_negotiation.cc:769] Sending connection context
0808 04:08:56.399559 (+ 109us) client_negotiation.cc:240] Negotiation successful
0808 04:08:56.399770 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22073,"mutex_wait_us":33,"thread_start_us":22360,"threads_started":1}
I20260808 04:08:56.400678 2047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:56.369072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.2:43895 (local address 127.0.0.1:58228)
0808 04:08:56.390505 (+ 21433us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:56.390538 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0808 04:08:56.390684 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:08:56.393622 (+ 2938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:08:56.393626 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:08:56.393650 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:08:56.394054 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:56.394063 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:56.395624 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:56.395629 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:56.397605 (+ 1976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:56.397620 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:56.399336 (+ 1716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:56.399343 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:56.399528 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:56.399563 (+ 35us) client_negotiation.cc:769] Sending connection context
0808 04:08:56.399657 (+ 94us) client_negotiation.cc:240] Negotiation successful
0808 04:08:56.399809 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21240,"mutex_wait_us":25,"thread_start_us":23333,"threads_started":1}
I20260808 04:08:56.400710 2044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:56.369294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58228 (local address 127.5.83.2:43895)
0808 04:08:56.393161 (+ 23867us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:56.393168 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:56.393194 (+ 26us) server_negotiation.cc:407] Connection header received
0808 04:08:56.393283 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:56.393288 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:56.393454 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:56.393607 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:08:56.394208 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:56.395500 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:56.397766 (+ 2266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:56.399096 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:56.399222 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:56.399399 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:56.400215 (+ 816us) server_negotiation.cc:299] Negotiation successful
0808 04:08:56.400395 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23709,"thread_start_us":23599,"threads_started":1}
I20260808 04:08:56.401396 2048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:56.371971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48502 (local address 127.5.83.3:41775)
0808 04:08:56.392198 (+ 20227us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:56.392207 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:56.392818 (+ 611us) server_negotiation.cc:407] Connection header received
0808 04:08:56.393122 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:56.393128 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:56.393315 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:56.393487 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:08:56.394618 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:56.396265 (+ 1647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:56.398348 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:56.399658 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:56.399796 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:56.400015 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:56.400942 (+ 927us) server_negotiation.cc:299] Negotiation successful
0808 04:08:56.401092 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20068,"thread_start_us":19924,"threads_started":1}
I20260808 04:08:56.401854 2040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:56.371647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.3:41775 (local address 127.0.0.1:48502)
0808 04:08:56.392704 (+ 21057us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:56.392722 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 04:08:56.392925 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:08:56.393792 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:08:56.393796 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:08:56.393821 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:08:56.394411 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:56.394426 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:56.396417 (+ 1991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:56.396422 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:56.398201 (+ 1779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:56.398212 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:56.400185 (+ 1973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:56.400192 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:56.400503 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:56.400537 (+ 34us) client_negotiation.cc:769] Sending connection context
0808 04:08:56.400629 (+ 92us) client_negotiation.cc:240] Negotiation successful
0808 04:08:56.401589 (+ 960us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20922,"thread_start_us":18549,"threads_started":1}
I20260808 04:08:56.404161 1735 tablet_service.cc:1467] Processing CreateTablet for tablet e7788910c0c84a458492e203aa2744c7 (DEFAULT_TABLE table=test-table [id=898da697915a48bfbdb73f8fb8ef1f2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:08:56.404155 1527 tablet_service.cc:1467] Processing CreateTablet for tablet e7788910c0c84a458492e203aa2744c7 (DEFAULT_TABLE table=test-table [id=898da697915a48bfbdb73f8fb8ef1f2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:08:56.404841 1926 tablet_service.cc:1467] Processing CreateTablet for tablet e7788910c0c84a458492e203aa2744c7 (DEFAULT_TABLE table=test-table [id=898da697915a48bfbdb73f8fb8ef1f2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:08:56.405627 1735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7788910c0c84a458492e203aa2744c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:08:56.405627 1527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7788910c0c84a458492e203aa2744c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:08:56.405923 1926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7788910c0c84a458492e203aa2744c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:08:56.416803 2049 tablet_bootstrap.cc:492] T e7788910c0c84a458492e203aa2744c7 P da20ac16c210419bbc22b151f1e10239: Bootstrap starting.
I20260808 04:08:56.419396 2051 tablet_bootstrap.cc:492] T e7788910c0c84a458492e203aa2744c7 P 0e416e8debae4680af6860cf9d0dce7f: Bootstrap starting.
I20260808 04:08:56.419909 2049 tablet_bootstrap.cc:654] T e7788910c0c84a458492e203aa2744c7 P da20ac16c210419bbc22b151f1e10239: Neither blocks nor log segments found. Creating new log.
I20260808 04:08:56.421252 2049 log.cc:824] T e7788910c0c84a458492e203aa2744c7 P da20ac16c210419bbc22b151f1e10239: Log is configured to *not* fsync() on all Append() calls
I20260808 04:08:56.422565 2055 tablet_bootstrap.cc:492] T e7788910c0c84a458492e203aa2744c7 P 0a9d391ff4724d4aa535d40a05a03a34: Bootstrap starting.
I20260808 04:08:56.422761 2051 tablet_bootstrap.cc:654] T e7788910c0c84a458492e203aa2744c7 P 0e416e8debae4680af6860cf9d0dce7f: Neither blocks nor log segments found. Creating new log.
I20260808 04:08:56.423971 2051 log.cc:824] T e7788910c0c84a458492e203aa2744c7 P 0e416e8debae4680af6860cf9d0dce7f: Log is configured to *not* fsync() on all Append() calls
I20260808 04:08:56.425168 2049 tablet_bootstrap.cc:492] T e7788910c0c84a458492e203aa2744c7 P da20ac16c210419bbc22b151f1e10239: No bootstrap required, opened a new log
I20260808 04:08:56.425432 2049 ts_tablet_manager.cc:1397] T e7788910c0c84a458492e203aa2744c7 P da20ac16c210419bbc22b151f1e10239: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260808 04:08:56.425800 2055 tablet_bootstrap.cc:654] T e7788910c0c84a458492e203aa2744c7 P 0a9d391ff4724d4aa535d40a05a03a34: Neither blocks nor log segments found. Creating new log.
I20260808 04:08:56.426959 2055 log.cc:824] T e7788910c0c84a458492e203aa2744c7 P 0a9d391ff4724d4aa535d40a05a03a34: Log is configured to *not* fsync() on all Append() calls
I20260808 04:08:56.427134 2051 tablet_bootstrap.cc:492] T e7788910c0c84a458492e203aa2744c7 P 0e416e8debae4680af6860cf9d0dce7f: No bootstrap required, opened a new log
I20260808 04:08:56.427374 2051 ts_tablet_manager.cc:1397] T e7788910c0c84a458492e203aa2744c7 P 0e416e8debae4680af6860cf9d0dce7f: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260808 04:08:56.429750 2055 tablet_bootstrap.cc:492] T e7788910c0c84a458492e203aa2744c7 P 0a9d391ff4724d4aa535d40a05a03a34: No bootstrap required, opened a new log
I20260808 04:08:56.430009 2055 ts_tablet_manager.cc:1397] T e7788910c0c84a458492e203aa2744c7 P 0a9d391ff4724d4aa535d40a05a03a34: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260808 04:08:56.432808 2049 raft_consensus.cc:348] T e7788910c0c84a458492e203aa2744c7 P da20ac16c210419bbc22b151f1e10239 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e416e8debae4680af6860cf9d0dce7f" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 43895 } } peers { permanent_uuid: "0a9d391ff4724d4aa535d40a05a03a34" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 39401 } } peers { permanent_uuid: "da20ac16c210419bbc22b151f1e10239" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 41775 } }
I20260808 04:08:56.433127 2049 raft_consensus.cc:374] T e7788910c0c84a458492e203aa2744c7 P da20ac16c210419bbc22b151f1e10239 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:08:56.433228 2049 raft_consensus.cc:729] T e7788910c0c84a458492e203aa2744c7 P da20ac16c210419bbc22b151f1e10239 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da20ac16c210419bbc22b151f1e10239, State: Initialized, Role: FOLLOWER
I20260808 04:08:56.433770 2049 consensus_queue.cc:260] T e7788910c0c84a458492e203aa2744c7 P da20ac16c210419bbc22b151f1e10239 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e416e8debae4680af6860cf9d0dce7f" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 43895 } } peers { permanent_uuid: "0a9d391ff4724d4aa535d40a05a03a34" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 39401 } } peers { permanent_uuid: "da20ac16c210419bbc22b151f1e10239" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 41775 } }
I20260808 04:08:56.434762 2023 heartbeater.cc:503] Master 127.5.83.62:45587 was elected leader, sending a full tablet report...
I20260808 04:08:56.436726 2049 ts_tablet_manager.cc:1428] T e7788910c0c84a458492e203aa2744c7 P da20ac16c210419bbc22b151f1e10239: Time spent starting tablet: real 0.011s user 0.009s sys 0.001s
I20260808 04:08:56.437271 2051 raft_consensus.cc:348] T e7788910c0c84a458492e203aa2744c7 P 0e416e8debae4680af6860cf9d0dce7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e416e8debae4680af6860cf9d0dce7f" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 43895 } } peers { permanent_uuid: "0a9d391ff4724d4aa535d40a05a03a34" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 39401 } } peers { permanent_uuid: "da20ac16c210419bbc22b151f1e10239" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 41775 } }
I20260808 04:08:56.437701 2051 raft_consensus.cc:374] T e7788910c0c84a458492e203aa2744c7 P 0e416e8debae4680af6860cf9d0dce7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:08:56.437809 2051 raft_consensus.cc:729] T e7788910c0c84a458492e203aa2744c7 P 0e416e8debae4680af6860cf9d0dce7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e416e8debae4680af6860cf9d0dce7f, State: Initialized, Role: FOLLOWER
I20260808 04:08:56.438385 2051 consensus_queue.cc:260] T e7788910c0c84a458492e203aa2744c7 P 0e416e8debae4680af6860cf9d0dce7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e416e8debae4680af6860cf9d0dce7f" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 43895 } } peers { permanent_uuid: "0a9d391ff4724d4aa535d40a05a03a34" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 39401 } } peers { permanent_uuid: "da20ac16c210419bbc22b151f1e10239" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 41775 } }
I20260808 04:08:56.439888 1822 heartbeater.cc:503] Master 127.5.83.62:45587 was elected leader, sending a full tablet report...
I20260808 04:08:56.440186 2051 ts_tablet_manager.cc:1428] T e7788910c0c84a458492e203aa2744c7 P 0e416e8debae4680af6860cf9d0dce7f: Time spent starting tablet: real 0.013s user 0.004s sys 0.009s
I20260808 04:08:56.440781 2055 raft_consensus.cc:348] T e7788910c0c84a458492e203aa2744c7 P 0a9d391ff4724d4aa535d40a05a03a34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e416e8debae4680af6860cf9d0dce7f" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 43895 } } peers { permanent_uuid: "0a9d391ff4724d4aa535d40a05a03a34" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 39401 } } peers { permanent_uuid: "da20ac16c210419bbc22b151f1e10239" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 41775 } }
I20260808 04:08:56.441092 2055 raft_consensus.cc:374] T e7788910c0c84a458492e203aa2744c7 P 0a9d391ff4724d4aa535d40a05a03a34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:08:56.441195 2055 raft_consensus.cc:729] T e7788910c0c84a458492e203aa2744c7 P 0a9d391ff4724d4aa535d40a05a03a34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a9d391ff4724d4aa535d40a05a03a34, State: Initialized, Role: FOLLOWER
I20260808 04:08:56.441730 2055 consensus_queue.cc:260] T e7788910c0c84a458492e203aa2744c7 P 0a9d391ff4724d4aa535d40a05a03a34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e416e8debae4680af6860cf9d0dce7f" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 43895 } } peers { permanent_uuid: "0a9d391ff4724d4aa535d40a05a03a34" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 39401 } } peers { permanent_uuid: "da20ac16c210419bbc22b151f1e10239" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 41775 } }
I20260808 04:08:56.443218 2055 ts_tablet_manager.cc:1428] T e7788910c0c84a458492e203aa2744c7 P 0a9d391ff4724d4aa535d40a05a03a34: Time spent starting tablet: real 0.013s user 0.007s sys 0.006s
W20260808 04:08:56.552973 2024 tablet.cc:2369] T e7788910c0c84a458492e203aa2744c7 P da20ac16c210419bbc22b151f1e10239: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 04:08:56.560602 1638 tablet.cc:2369] T e7788910c0c84a458492e203aa2744c7 P 0a9d391ff4724d4aa535d40a05a03a34: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 04:08:56.562794 1823 tablet.cc:2369] T e7788910c0c84a458492e203aa2744c7 P 0e416e8debae4680af6860cf9d0dce7f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:08:56.585469 2062 raft_consensus.cc:482] T e7788910c0c84a458492e203aa2744c7 P 0a9d391ff4724d4aa535d40a05a03a34 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:08:56.585719 2062 raft_consensus.cc:504] T e7788910c0c84a458492e203aa2744c7 P 0a9d391ff4724d4aa535d40a05a03a34 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e416e8debae4680af6860cf9d0dce7f" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 43895 } } peers { permanent_uuid: "0a9d391ff4724d4aa535d40a05a03a34" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 39401 } } peers { permanent_uuid: "da20ac16c210419bbc22b151f1e10239" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 41775 } }
I20260808 04:08:56.587267 2062 leader_election.cc:288] T e7788910c0c84a458492e203aa2744c7 P 0a9d391ff4724d4aa535d40a05a03a34 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e416e8debae4680af6860cf9d0dce7f (127.5.83.2:43895), da20ac16c210419bbc22b151f1e10239 (127.5.83.3:41775)
I20260808 04:08:56.593360 2048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:56.587656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.1:39915 (local address 127.5.83.3:41775)
0808 04:08:56.587889 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:56.587895 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:56.588382 (+ 487us) server_negotiation.cc:407] Connection header received
0808 04:08:56.588596 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:56.588601 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:56.588660 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:56.588733 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:08:56.589272 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:56.590003 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:56.591740 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:56.592508 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:56.592623 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:56.592724 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:56.592997 (+ 273us) server_negotiation.cc:299] Negotiation successful
0808 04:08:56.593118 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260808 04:08:56.593483 2063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:56.587593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.3:41775 (local address 127.5.83.1:39915)
0808 04:08:56.588276 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:56.588296 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 04:08:56.588454 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:08:56.588769 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:08:56.588773 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:08:56.588799 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:08:56.589134 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:56.589143 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:56.590126 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:56.590131 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:56.591562 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:56.591575 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:56.592704 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:56.592711 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:56.592844 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:56.592869 (+ 25us) client_negotiation.cc:769] Sending connection context
0808 04:08:56.592953 (+ 84us) client_negotiation.cc:240] Negotiation successful
0808 04:08:56.593082 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"mutex_wait_us":1,"thread_start_us":347,"threads_started":1}
I20260808 04:08:56.593756 2044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:56.587622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.1:51907 (local address 127.5.83.2:43895)
0808 04:08:56.587850 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:56.587857 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:56.588792 (+ 935us) server_negotiation.cc:407] Connection header received
0808 04:08:56.589062 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:56.589066 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:56.589125 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:56.589200 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:08:56.589706 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:56.590411 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:56.592128 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:56.592877 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:56.593001 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:56.593104 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:56.593382 (+ 278us) server_negotiation.cc:299] Negotiation successful
0808 04:08:56.593516 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 04:08:56.594113 2065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:56.587664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.2:43895 (local address 127.5.83.1:51907)
0808 04:08:56.588678 (+ 1014us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:56.588697 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 04:08:56.588913 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:08:56.589238 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:08:56.589243 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:08:56.589266 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:08:56.589551 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:56.589560 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:56.590559 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:56.590563 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:56.591974 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:56.591986 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:56.593105 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:56.593112 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:56.593237 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:56.593260 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 04:08:56.593348 (+ 88us) client_negotiation.cc:240] Negotiation successful
0808 04:08:56.593778 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":918,"spinlock_wait_cycles":22784,"thread_start_us":509,"threads_started":1}
I20260808 04:08:56.594376 1953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7788910c0c84a458492e203aa2744c7" candidate_uuid: "0a9d391ff4724d4aa535d40a05a03a34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da20ac16c210419bbc22b151f1e10239" is_pre_election: true
I20260808 04:08:56.594830 1953 raft_consensus.cc:2454] T e7788910c0c84a458492e203aa2744c7 P da20ac16c210419bbc22b151f1e10239 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a9d391ff4724d4aa535d40a05a03a34 in term 0.
I20260808 04:08:56.594949 1765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7788910c0c84a458492e203aa2744c7" candidate_uuid: "0a9d391ff4724d4aa535d40a05a03a34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e416e8debae4680af6860cf9d0dce7f" is_pre_election: true
I20260808 04:08:56.595456 1765 raft_consensus.cc:2454] T e7788910c0c84a458492e203aa2744c7 P 0e416e8debae4680af6860cf9d0dce7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a9d391ff4724d4aa535d40a05a03a34 in term 0.
I20260808 04:08:56.595474 1465 leader_election.cc:302] T e7788910c0c84a458492e203aa2744c7 P 0a9d391ff4724d4aa535d40a05a03a34 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a9d391ff4724d4aa535d40a05a03a34, da20ac16c210419bbc22b151f1e10239; no voters:
I20260808 04:08:56.595851 2062 raft_consensus.cc:2781] T e7788910c0c84a458492e203aa2744c7 P 0a9d391ff4724d4aa535d40a05a03a34 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:08:56.596014 2062 raft_consensus.cc:482] T e7788910c0c84a458492e203aa2744c7 P 0a9d391ff4724d4aa535d40a05a03a34 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:08:56.596127 2062 raft_consensus.cc:3037] T e7788910c0c84a458492e203aa2744c7 P 0a9d391ff4724d4aa535d40a05a03a34 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:08:56.598342 2062 raft_consensus.cc:504] T e7788910c0c84a458492e203aa2744c7 P 0a9d391ff4724d4aa535d40a05a03a34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e416e8debae4680af6860cf9d0dce7f" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 43895 } } peers { permanent_uuid: "0a9d391ff4724d4aa535d40a05a03a34" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 39401 } } peers { permanent_uuid: "da20ac16c210419bbc22b151f1e10239" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 41775 } }
I20260808 04:08:56.598914 2062 leader_election.cc:288] T e7788910c0c84a458492e203aa2744c7 P 0a9d391ff4724d4aa535d40a05a03a34 [CANDIDATE]: Term 1 election: Requested vote from peers 0e416e8debae4680af6860cf9d0dce7f (127.5.83.2:43895), da20ac16c210419bbc22b151f1e10239 (127.5.83.3:41775)
I20260808 04:08:56.599294 1765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7788910c0c84a458492e203aa2744c7" candidate_uuid: "0a9d391ff4724d4aa535d40a05a03a34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e416e8debae4680af6860cf9d0dce7f"
I20260808 04:08:56.599373 1953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7788910c0c84a458492e203aa2744c7" candidate_uuid: "0a9d391ff4724d4aa535d40a05a03a34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da20ac16c210419bbc22b151f1e10239"
I20260808 04:08:56.599509 1765 raft_consensus.cc:3037] T e7788910c0c84a458492e203aa2744c7 P 0e416e8debae4680af6860cf9d0dce7f [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:08:56.599555 1953 raft_consensus.cc:3037] T e7788910c0c84a458492e203aa2744c7 P da20ac16c210419bbc22b151f1e10239 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:08:56.602756 1953 raft_consensus.cc:2454] T e7788910c0c84a458492e203aa2744c7 P da20ac16c210419bbc22b151f1e10239 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a9d391ff4724d4aa535d40a05a03a34 in term 1.
I20260808 04:08:56.602756 1765 raft_consensus.cc:2454] T e7788910c0c84a458492e203aa2744c7 P 0e416e8debae4680af6860cf9d0dce7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a9d391ff4724d4aa535d40a05a03a34 in term 1.
I20260808 04:08:56.603145 1470 leader_election.cc:302] T e7788910c0c84a458492e203aa2744c7 P 0a9d391ff4724d4aa535d40a05a03a34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a9d391ff4724d4aa535d40a05a03a34, 0e416e8debae4680af6860cf9d0dce7f; no voters:
I20260808 04:08:56.603425 2062 raft_consensus.cc:2781] T e7788910c0c84a458492e203aa2744c7 P 0a9d391ff4724d4aa535d40a05a03a34 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:08:56.604743 2062 raft_consensus.cc:686] T e7788910c0c84a458492e203aa2744c7 P 0a9d391ff4724d4aa535d40a05a03a34 [term 1 LEADER]: Becoming Leader. State: Replica: 0a9d391ff4724d4aa535d40a05a03a34, State: Running, Role: LEADER
I20260808 04:08:56.605253 2062 consensus_queue.cc:237] T e7788910c0c84a458492e203aa2744c7 P 0a9d391ff4724d4aa535d40a05a03a34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e416e8debae4680af6860cf9d0dce7f" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 43895 } } peers { permanent_uuid: "0a9d391ff4724d4aa535d40a05a03a34" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 39401 } } peers { permanent_uuid: "da20ac16c210419bbc22b151f1e10239" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 41775 } }
I20260808 04:08:56.609730 1344 catalog_manager.cc:5495] T e7788910c0c84a458492e203aa2744c7 P 0a9d391ff4724d4aa535d40a05a03a34 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a9d391ff4724d4aa535d40a05a03a34 (127.5.83.1). New cstate: current_term: 1 leader_uuid: "0a9d391ff4724d4aa535d40a05a03a34" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e416e8debae4680af6860cf9d0dce7f" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 43895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a9d391ff4724d4aa535d40a05a03a34" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 39401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da20ac16c210419bbc22b151f1e10239" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 41775 } health_report { overall_health: UNKNOWN } } }
I20260808 04:08:56.630776 2046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:56.625549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48076 (local address 127.5.83.1:39401)
0808 04:08:56.625736 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:56.625741 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:56.625779 (+ 38us) server_negotiation.cc:407] Connection header received
0808 04:08:56.626004 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:56.626007 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:56.626082 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:56.626221 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:08:56.626610 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:56.627319 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:56.628721 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:56.629115 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:56.629224 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:56.629721 (+ 497us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:08:56.630210 (+ 489us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:08:56.630353 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:56.630447 (+ 94us) server_negotiation.cc:299] Negotiation successful
0808 04:08:56.630526 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 04:08:56.638689 1953 raft_consensus.cc:1261] T e7788910c0c84a458492e203aa2744c7 P da20ac16c210419bbc22b151f1e10239 [term 1 FOLLOWER]: Refusing update from remote peer 0a9d391ff4724d4aa535d40a05a03a34: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:08:56.638712 1765 raft_consensus.cc:1261] T e7788910c0c84a458492e203aa2744c7 P 0e416e8debae4680af6860cf9d0dce7f [term 1 FOLLOWER]: Refusing update from remote peer 0a9d391ff4724d4aa535d40a05a03a34: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:08:56.639591 2062 consensus_queue.cc:1035] T e7788910c0c84a458492e203aa2744c7 P 0a9d391ff4724d4aa535d40a05a03a34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da20ac16c210419bbc22b151f1e10239" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:08:56.639896 2068 consensus_queue.cc:1035] T e7788910c0c84a458492e203aa2744c7 P 0a9d391ff4724d4aa535d40a05a03a34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e416e8debae4680af6860cf9d0dce7f" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 43895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:08:56.650374 2073 mvcc.cc:205] Tried to move back new op lower bound from 7316120111663022080 to 7316120111538319360. Current Snapshot: MvccSnapshot[applied={T|T < 7316120111663022080}]
I20260808 04:08:56.652794 2075 mvcc.cc:205] Tried to move back new op lower bound from 7316120111663022080 to 7316120111538319360. Current Snapshot: MvccSnapshot[applied={T|T < 7316120111663022080}]
I20260808 04:08:56.652818 2071 mvcc.cc:205] Tried to move back new op lower bound from 7316120111663022080 to 7316120111538319360. Current Snapshot: MvccSnapshot[applied={T|T < 7316120111663022080}]
I20260808 04:08:56.662693 1344 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:08:56.662935 1344 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:08:56.666179 1344 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9504ad9897c841e8aa2f0a5442f1de68: Sending DeleteTablet for 3 replicas of tablet e7788910c0c84a458492e203aa2744c7
I20260808 04:08:56.667172 1735 tablet_service.cc:1514] Processing DeleteTablet for tablet e7788910c0c84a458492e203aa2744c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:08:56 UTC) from {username='kudu', principal='kudu/127.5.83.62@KRBTEST.COM'} at 127.0.0.1:58228
I20260808 04:08:56.667279 1527 tablet_service.cc:1514] Processing DeleteTablet for tablet e7788910c0c84a458492e203aa2744c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:08:56 UTC) from {username='kudu', principal='kudu/127.5.83.62@KRBTEST.COM'} at 127.0.0.1:48070
I20260808 04:08:56.667595 1926 tablet_service.cc:1514] Processing DeleteTablet for tablet e7788910c0c84a458492e203aa2744c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:08:56 UTC) from {username='kudu', principal='kudu/127.5.83.62@KRBTEST.COM'} at 127.0.0.1:48502
I20260808 04:08:56.669634 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 1442
I20260808 04:08:56.676347 2085 tablet_replica.cc:330] T e7788910c0c84a458492e203aa2744c7 P da20ac16c210419bbc22b151f1e10239: stopping tablet replica
I20260808 04:08:56.676769 2085 raft_consensus.cc:2229] T e7788910c0c84a458492e203aa2744c7 P da20ac16c210419bbc22b151f1e10239 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:08:56.677276 2085 raft_consensus.cc:2258] T e7788910c0c84a458492e203aa2744c7 P da20ac16c210419bbc22b151f1e10239 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:08:56.690384 2085 ts_tablet_manager.cc:1905] T e7788910c0c84a458492e203aa2744c7 P da20ac16c210419bbc22b151f1e10239: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:08:56.691395 2081 tablet_replica.cc:330] T e7788910c0c84a458492e203aa2744c7 P 0e416e8debae4680af6860cf9d0dce7f: stopping tablet replica
I20260808 04:08:56.691879 2081 raft_consensus.cc:2229] T e7788910c0c84a458492e203aa2744c7 P 0e416e8debae4680af6860cf9d0dce7f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:08:56.692378 2084 tablet_replica.cc:330] T e7788910c0c84a458492e203aa2744c7 P 0a9d391ff4724d4aa535d40a05a03a34: stopping tablet replica
I20260808 04:08:56.692378 2081 raft_consensus.cc:2258] T e7788910c0c84a458492e203aa2744c7 P 0e416e8debae4680af6860cf9d0dce7f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:08:56.692716 2084 raft_consensus.cc:2229] T e7788910c0c84a458492e203aa2744c7 P 0a9d391ff4724d4aa535d40a05a03a34 [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:08:56.693360 2084 raft_consensus.cc:2258] T e7788910c0c84a458492e203aa2744c7 P 0a9d391ff4724d4aa535d40a05a03a34 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:08:56.694697 2081 ts_tablet_manager.cc:1905] T e7788910c0c84a458492e203aa2744c7 P 0e416e8debae4680af6860cf9d0dce7f: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 08 04:08:56 dist-test-slave-1l3l krb5kdc[1285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162134, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.1@KRBTEST.COM
I20260808 04:08:56.697147 2084 ts_tablet_manager.cc:1905] T e7788910c0c84a458492e203aa2744c7 P 0a9d391ff4724d4aa535d40a05a03a34: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:08:56.701817 2085 ts_tablet_manager.cc:1918] T e7788910c0c84a458492e203aa2744c7 P da20ac16c210419bbc22b151f1e10239: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:08:56.701839 2081 ts_tablet_manager.cc:1918] T e7788910c0c84a458492e203aa2744c7 P 0e416e8debae4680af6860cf9d0dce7f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:08:56.701964 2085 log.cc:1196] T e7788910c0c84a458492e203aa2744c7 P da20ac16c210419bbc22b151f1e10239: Deleting WAL directory at /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/minicluster-data/ts-2/wal/wals/e7788910c0c84a458492e203aa2744c7
I20260808 04:08:56.702131 2081 log.cc:1196] T e7788910c0c84a458492e203aa2744c7 P 0e416e8debae4680af6860cf9d0dce7f: Deleting WAL directory at /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/minicluster-data/ts-1/wal/wals/e7788910c0c84a458492e203aa2744c7
I20260808 04:08:56.702448 2085 ts_tablet_manager.cc:1939] T e7788910c0c84a458492e203aa2744c7 P da20ac16c210419bbc22b151f1e10239: Deleting consensus metadata
I20260808 04:08:56.702778 2081 ts_tablet_manager.cc:1939] T e7788910c0c84a458492e203aa2744c7 P 0e416e8debae4680af6860cf9d0dce7f: Deleting consensus metadata
I20260808 04:08:56.704288 1323 catalog_manager.cc:4841] TS da20ac16c210419bbc22b151f1e10239 (127.5.83.3:41775): tablet e7788910c0c84a458492e203aa2744c7 (table test-table [id=898da697915a48bfbdb73f8fb8ef1f2c]) successfully deleted
I20260808 04:08:56.704486 1328 catalog_manager.cc:4841] TS 0e416e8debae4680af6860cf9d0dce7f (127.5.83.2:43895): tablet e7788910c0c84a458492e203aa2744c7 (table test-table [id=898da697915a48bfbdb73f8fb8ef1f2c]) successfully deleted
I20260808 04:08:56.704929 2084 ts_tablet_manager.cc:1918] T e7788910c0c84a458492e203aa2744c7 P 0a9d391ff4724d4aa535d40a05a03a34: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:08:56.705085 2084 log.cc:1196] T e7788910c0c84a458492e203aa2744c7 P 0a9d391ff4724d4aa535d40a05a03a34: Deleting WAL directory at /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161826605655-5452-0/minicluster-data/ts-0/wal/wals/e7788910c0c84a458492e203aa2744c7
I20260808 04:08:56.705525 2084 ts_tablet_manager.cc:1939] T e7788910c0c84a458492e203aa2744c7 P 0a9d391ff4724d4aa535d40a05a03a34: Deleting consensus metadata
I20260808 04:08:56.706905 1325 catalog_manager.cc:4841] TS 0a9d391ff4724d4aa535d40a05a03a34 (127.5.83.1:39401): tablet e7788910c0c84a458492e203aa2744c7 (table test-table [id=898da697915a48bfbdb73f8fb8ef1f2c]) successfully deleted
I20260808 04:08:56.711104 2046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:56.670922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48086 (local address 127.5.83.1:39401)
0808 04:08:56.671101 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:56.671107 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:56.682919 (+ 11812us) server_negotiation.cc:407] Connection header received
0808 04:08:56.683354 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:56.683360 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:56.683467 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:56.683548 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:08:56.685446 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:56.686815 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:56.690904 (+ 4089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:56.691447 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:56.691569 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:56.703883 (+ 12314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:08:56.703896 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:08:56.703912 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:08:56.703968 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:08:56.708105 (+ 4137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:08:56.708932 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:08:56.708941 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:08:56.708948 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:08:56.709093 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:08:56.709502 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:08:56.709507 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:08:56.709509 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:08:56.709938 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:08:56.710209 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:56.710536 (+ 327us) server_negotiation.cc:299] Negotiation successful
0808 04:08:56.710762 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 04:08:56.711778 1630 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:08:56.809558 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 1442
I20260808 04:08:56.833585 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 1647
Aug 08 04:08:56 dist-test-slave-1l3l krb5kdc[1285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162134, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.2@KRBTEST.COM
I20260808 04:08:56.848389 2044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:56.834312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58242 (local address 127.5.83.2:43895)
0808 04:08:56.834468 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:56.834473 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:56.834552 (+ 79us) server_negotiation.cc:407] Connection header received
0808 04:08:56.834689 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:56.834693 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:56.834775 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:56.834860 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:08:56.836223 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:56.837013 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:56.838204 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:56.838602 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:56.838712 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:56.842863 (+ 4151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:08:56.842877 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:08:56.842891 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:08:56.842925 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:08:56.846002 (+ 3077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:08:56.846609 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:08:56.846615 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:08:56.846617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:08:56.846729 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:08:56.847008 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:08:56.847011 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:08:56.847012 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:08:56.847357 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:08:56.847602 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:56.847869 (+ 267us) server_negotiation.cc:299] Negotiation successful
0808 04:08:56.848022 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 04:08:56.849010 1817 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:08:56.934593 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 1647
I20260808 04:08:56.954783 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 1828
Aug 08 04:08:56 dist-test-slave-1l3l krb5kdc[1285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162134, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.3@KRBTEST.COM
I20260808 04:08:56.969625 2048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:56.955545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48510 (local address 127.5.83.3:41775)
0808 04:08:56.955740 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:56.955746 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:56.955767 (+ 21us) server_negotiation.cc:407] Connection header received
0808 04:08:56.955968 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:56.955972 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:56.956038 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:56.956150 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:08:56.957546 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:56.958293 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:56.959569 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:56.960021 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:56.960138 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:56.964377 (+ 4239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:08:56.964393 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:08:56.964415 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:08:56.964447 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:08:56.966821 (+ 2374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:08:56.967445 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:08:56.967452 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:08:56.967454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:08:56.967598 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:08:56.968153 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:08:56.968158 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:08:56.968162 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:08:56.968508 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:08:56.968794 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:56.969106 (+ 312us) server_negotiation.cc:299] Negotiation successful
0808 04:08:56.969317 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 04:08:56.970221 2014 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:08:57.058486 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 1828
I20260808 04:08:57.078999 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 1300
I20260808 04:08:57.080031 1387 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:08:57.186208 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 1300
2026-08-08T04:08:57Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4366 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-08T04:08:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:08:57Z Disabled control of system clock
I20260808 04:08:57.276466 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161826605655-5452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161826605655-5452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161826605655-5452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:37889
--webserver_interface=127.5.83.62
--webserver_port=0
--builtin_ntp_servers=127.5.83.20:35447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.62:37889
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 04:08:57.590528 2099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:08:57.590790 2099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:08:57.590909 2099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:08:57.598001 2099 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:08:57.598099 2099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:08:57.598157 2099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:08:57.598240 2099 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:08:57.598294 2099 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:08:57.607213 2099 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:35447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161826605655-5452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161826605655-5452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.62:37889
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:37889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161826605655-5452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:08:57.608523 2099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:08:57.610246 2099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:08:57.621136 2106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:08:57.621186 2111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:08:57.621150 2108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:08:57.622449 2099 server_base.cc:1034] running on GCE node
I20260808 04:08:57.623152 2099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:08:57.624509 2099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:08:57.625705 2099 hybrid_clock.cc:648] HybridClock initialized: now 1786162137625657 us; error 39 us; skew 500 ppm
I20260808 04:08:57.626011 2099 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 04:08:57.628211 2099 webserver.cc:460] Webserver started at http://127.5.83.62:43759/ using document root <none> and password file <none>
I20260808 04:08:57.628844 2099 fs_manager.cc:359] Metadata directory not provided
I20260808 04:08:57.628947 2099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:08:57.629252 2099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:08:57.632153 2099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161826605655-5452-0/minicluster-data/master-0/data/instance:
uuid: "eb9e4100dee94a3687369cfbbafacece"
format_stamp: "Formatted at 2026-08-08 04:08:57 on dist-test-slave-1l3l"
server_key: "1bb5444ceb09589984a65912a3ac1c10"
server_key_iv: "90f13e778ce0f24ca61eafed24be3a93"
server_key_version: "encryptionkey@0"
I20260808 04:08:57.632977 2099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161826605655-5452-0/minicluster-data/master-0/wal/instance:
uuid: "eb9e4100dee94a3687369cfbbafacece"
format_stamp: "Formatted at 2026-08-08 04:08:57 on dist-test-slave-1l3l"
server_key: "1bb5444ceb09589984a65912a3ac1c10"
server_key_iv: "90f13e778ce0f24ca61eafed24be3a93"
server_key_version: "encryptionkey@0"
I20260808 04:08:57.638043 2099 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 04:08:57.641254 2122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:08:57.642423 2099 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 04:08:57.642585 2099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161826605655-5452-0/minicluster-data/master-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161826605655-5452-0/minicluster-data/master-0/wal
uuid: "eb9e4100dee94a3687369cfbbafacece"
format_stamp: "Formatted at 2026-08-08 04:08:57 on dist-test-slave-1l3l"
server_key: "1bb5444ceb09589984a65912a3ac1c10"
server_key_iv: "90f13e778ce0f24ca61eafed24be3a93"
server_key_version: "encryptionkey@0"
I20260808 04:08:57.642777 2099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161826605655-5452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161826605655-5452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161826605655-5452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:08:57.654565 2099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:08:57.655269 2099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:08:57.655544 2099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:08:57.672437 2099 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.62:37889
I20260808 04:08:57.672456 2201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.62:37889 every 8 connection(s)
I20260808 04:08:57.673655 2099 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
I20260808 04:08:57.677168 2203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:08:57.677491 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 2099
I20260808 04:08:57.677644 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161826605655-5452-0/minicluster-data/master-0/wal/instance
I20260808 04:08:57.678107 5452 external_mini_cluster.cc:1428] Setting key 319f6e66c12372b3ae8c73388986363a
I20260808 04:08:57.690711 2203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb9e4100dee94a3687369cfbbafacece: Bootstrap starting.
I20260808 04:08:57.693527 2203 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb9e4100dee94a3687369cfbbafacece: Neither blocks nor log segments found. Creating new log.
I20260808 04:08:57.694554 2203 log.cc:824] T 00000000000000000000000000000000 P eb9e4100dee94a3687369cfbbafacece: Log is configured to *not* fsync() on all Append() calls
I20260808 04:08:57.697326 2211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:57.679855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48868 (local address 127.5.83.62:37889)
0808 04:08:57.680941 (+ 1086us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:57.680955 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:57.681008 (+ 53us) server_negotiation.cc:407] Connection header received
0808 04:08:57.683369 (+ 2361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:57.683396 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:57.683978 (+ 582us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:57.684413 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:08:57.684911 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:57.686682 (+ 1771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:57.687985 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:57.689029 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:57.689190 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:57.694644 (+ 5454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:08:57.694656 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:08:57.694673 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:08:57.695422 (+ 749us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:08:57.695504 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:08:57.695526 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:08:57.695615 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:57.695868 (+ 253us) server_negotiation.cc:299] Negotiation successful
0808 04:08:57.696180 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":329,"threads_started":1}
I20260808 04:08:57.697796 2203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb9e4100dee94a3687369cfbbafacece: No bootstrap required, opened a new log
I20260808 04:08:57.705032 2203 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb9e4100dee94a3687369cfbbafacece [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb9e4100dee94a3687369cfbbafacece" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 37889 } }
I20260808 04:08:57.705430 2203 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb9e4100dee94a3687369cfbbafacece [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:08:57.705528 2203 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb9e4100dee94a3687369cfbbafacece [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb9e4100dee94a3687369cfbbafacece, State: Initialized, Role: FOLLOWER
I20260808 04:08:57.706029 2203 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb9e4100dee94a3687369cfbbafacece [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb9e4100dee94a3687369cfbbafacece" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 37889 } }
I20260808 04:08:57.706243 2203 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb9e4100dee94a3687369cfbbafacece [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:08:57.706363 2203 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb9e4100dee94a3687369cfbbafacece [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:08:57.706526 2203 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb9e4100dee94a3687369cfbbafacece [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:08:57.708595 2203 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb9e4100dee94a3687369cfbbafacece [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb9e4100dee94a3687369cfbbafacece" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 37889 } }
I20260808 04:08:57.709028 2203 leader_election.cc:302] T 00000000000000000000000000000000 P eb9e4100dee94a3687369cfbbafacece [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb9e4100dee94a3687369cfbbafacece; no voters:
I20260808 04:08:57.709527 2203 leader_election.cc:288] T 00000000000000000000000000000000 P eb9e4100dee94a3687369cfbbafacece [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:08:57.709635 2213 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb9e4100dee94a3687369cfbbafacece [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:08:57.710239 2213 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb9e4100dee94a3687369cfbbafacece [term 1 LEADER]: Becoming Leader. State: Replica: eb9e4100dee94a3687369cfbbafacece, State: Running, Role: LEADER
I20260808 04:08:57.710745 2213 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb9e4100dee94a3687369cfbbafacece [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb9e4100dee94a3687369cfbbafacece" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 37889 } }
I20260808 04:08:57.711355 2203 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb9e4100dee94a3687369cfbbafacece [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:08:57.713423 2214 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb9e4100dee94a3687369cfbbafacece [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb9e4100dee94a3687369cfbbafacece" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb9e4100dee94a3687369cfbbafacece" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 37889 } } }
I20260808 04:08:57.713841 2214 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb9e4100dee94a3687369cfbbafacece [sys.catalog]: This master's current role is: LEADER
I20260808 04:08:57.715014 2216 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb9e4100dee94a3687369cfbbafacece [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb9e4100dee94a3687369cfbbafacece. Latest consensus state: current_term: 1 leader_uuid: "eb9e4100dee94a3687369cfbbafacece" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb9e4100dee94a3687369cfbbafacece" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 37889 } } }
I20260808 04:08:57.715310 2216 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb9e4100dee94a3687369cfbbafacece [sys.catalog]: This master's current role is: LEADER
I20260808 04:08:57.716780 2226 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:08:57.721540 2226 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:08:57.729714 2226 catalog_manager.cc:1347] Generated new cluster ID: 5ec181ee22ea486ea5bb2c597397b1aa
I20260808 04:08:57.729815 2226 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:08:57.742049 2226 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:08:57.743661 2226 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:08:57.757714 2226 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb9e4100dee94a3687369cfbbafacece: Generated new TSK 0
I20260808 04:08:57.758608 2226 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:08:57.769091 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161826605655-5452-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161826605655-5452-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161826605655-5452-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.1:0
--local_ip_for_outbound_sockets=127.5.83.1
--webserver_interface=127.5.83.1
--webserver_port=0
--tserver_master_addrs=127.5.83.62:37889
--builtin_ntp_servers=127.5.83.20:35447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 04:08:58.113056 2238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:08:58.113312 2238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:08:58.113440 2238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:08:58.120486 2238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:08:58.120570 2238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:08:58.120707 2238 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.1
I20260808 04:08:58.129267 2238 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:35447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161826605655-5452-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161826605655-5452-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.83.1
--webserver_port=0
--tserver_master_addrs=127.5.83.62:37889
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.1
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161826605655-5452-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:08:58.130398 2238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:08:58.132119 2238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:08:58.144697 2244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:08:58.144659 2246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:08:58.144734 2248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:08:58.145694 2238 server_base.cc:1034] running on GCE node
I20260808 04:08:58.146255 2238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:08:58.147152 2238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:08:58.148347 2238 hybrid_clock.cc:648] HybridClock initialized: now 1786162138148295 us; error 45 us; skew 500 ppm
I20260808 04:08:58.148633 2238 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 04:08:58.150651 2238 webserver.cc:460] Webserver started at http://127.5.83.1:38919/ using document root <none> and password file <none>
I20260808 04:08:58.151263 2238 fs_manager.cc:359] Metadata directory not provided
I20260808 04:08:58.151373 2238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:08:58.151671 2238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:08:58.154394 2238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161826605655-5452-0/minicluster-data/ts-0/data/instance:
uuid: "0efad64a49314c748e6e42fe4c19d2dd"
format_stamp: "Formatted at 2026-08-08 04:08:58 on dist-test-slave-1l3l"
server_key: "284c352f903922d5f047cbe037815774"
server_key_iv: "229510304884dbbb2e2e0dd62b476eaf"
server_key_version: "encryptionkey@0"
I20260808 04:08:58.155148 2238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161826605655-5452-0/minicluster-data/ts-0/wal/instance:
uuid: "0efad64a49314c748e6e42fe4c19d2dd"
format_stamp: "Formatted at 2026-08-08 04:08:58 on dist-test-slave-1l3l"
server_key: "284c352f903922d5f047cbe037815774"
server_key_iv: "229510304884dbbb2e2e0dd62b476eaf"
server_key_version: "encryptionkey@0"
I20260808 04:08:58.160066 2238 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260808 04:08:58.163216 2256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:08:58.164517 2238 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:08:58.164657 2238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161826605655-5452-0/minicluster-data/ts-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161826605655-5452-0/minicluster-data/ts-0/wal
uuid: "0efad64a49314c748e6e42fe4c19d2dd"
format_stamp: "Formatted at 2026-08-08 04:08:58 on dist-test-slave-1l3l"
server_key: "284c352f903922d5f047cbe037815774"
server_key_iv: "229510304884dbbb2e2e0dd62b476eaf"
server_key_version: "encryptionkey@0"
I20260808 04:08:58.164855 2238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161826605655-5452-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161826605655-5452-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161826605655-5452-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:08:58.176250 2238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:08:58.177018 2238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:08:58.177277 2238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:08:58.178157 2238 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:08:58.179565 2238 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:08:58.179663 2238 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:08:58.179845 2238 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:08:58.179920 2238 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:08:58.210069 2238 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.1:41573
I20260808 04:08:58.210103 2414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.1:41573 every 8 connection(s)
I20260808 04:08:58.211253 2238 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161826605655-5452-0/minicluster-data/ts-0/data/info.pb
I20260808 04:08:58.218842 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 2238
I20260808 04:08:58.219012 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161826605655-5452-0/minicluster-data/ts-0/wal/instance
I20260808 04:08:58.219513 5452 external_mini_cluster.cc:1428] Setting key 02661f05ba1308ffda6de1ca1dab7d5e
I20260808 04:08:58.222335 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161826605655-5452-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161826605655-5452-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161826605655-5452-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.2:0
--local_ip_for_outbound_sockets=127.5.83.2
--webserver_interface=127.5.83.2
--webserver_port=0
--tserver_master_addrs=127.5.83.62:37889
--builtin_ntp_servers=127.5.83.20:35447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 04:08:58.224035 2417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:58.213634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.1:53893 (local address 127.5.83.62:37889)
0808 04:08:58.214071 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:58.214075 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:58.215252 (+ 1177us) server_negotiation.cc:407] Connection header received
0808 04:08:58.217055 (+ 1803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:58.217060 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:58.217138 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:58.217236 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:08:58.218253 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:58.219233 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:58.221237 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:58.221773 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:58.221915 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:58.223247 (+ 1332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:08:58.223254 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:08:58.223258 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:08:58.223278 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:08:58.223324 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:08:58.223339 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:08:58.223426 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:58.223597 (+ 171us) server_negotiation.cc:299] Negotiation successful
0808 04:08:58.223685 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":223,"threads_started":1}
I20260808 04:08:58.224959 2418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:58.214224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:37889 (local address 127.5.83.1:53893)
0808 04:08:58.215115 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:58.215151 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0808 04:08:58.216794 (+ 1643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:08:58.217471 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:08:58.217479 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:08:58.217560 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:08:58.218081 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:58.218097 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:58.219386 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:58.219391 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:58.221085 (+ 1694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:58.221100 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:58.221975 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:58.221981 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:58.222264 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:58.222942 (+ 678us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:08:58.222945 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:08:58.222988 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:08:58.223058 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:08:58.223425 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:08:58.223432 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:08:58.223437 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 04:08:58.223567 (+ 130us) client_negotiation.cc:240] Negotiation successful
0808 04:08:58.223881 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":286,"threads_started":1}
I20260808 04:08:58.226284 2415 heartbeater.cc:347] Connected to a master server at 127.5.83.62:37889
I20260808 04:08:58.226537 2415 heartbeater.cc:464] Registering TS with master...
I20260808 04:08:58.227218 2415 heartbeater.cc:511] Master 127.5.83.62:37889 requested a full tablet report, sending...
I20260808 04:08:58.229112 2147 ts_manager.cc:194] Registered new tserver with Master: 0efad64a49314c748e6e42fe4c19d2dd (127.5.83.1:41573)
I20260808 04:08:58.230504 2147 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.83.1:53893
W20260808 04:08:58.550819 2419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:08:58.551091 2419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:08:58.551193 2419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:08:58.558091 2419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:08:58.558183 2419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:08:58.558291 2419 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.2
I20260808 04:08:58.566910 2419 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:35447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161826605655-5452-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161826605655-5452-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.83.2
--webserver_port=0
--tserver_master_addrs=127.5.83.62:37889
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.2
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161826605655-5452-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:08:58.568111 2419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:08:58.569850 2419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:08:58.582293 2429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:08:58.582288 2426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:08:58.582288 2425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:08:58.583115 2419 server_base.cc:1034] running on GCE node
I20260808 04:08:58.583681 2419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:08:58.584569 2419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:08:58.585764 2419 hybrid_clock.cc:648] HybridClock initialized: now 1786162138585720 us; error 39 us; skew 500 ppm
I20260808 04:08:58.586030 2419 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 04:08:58.588234 2419 webserver.cc:460] Webserver started at http://127.5.83.2:46171/ using document root <none> and password file <none>
I20260808 04:08:58.588829 2419 fs_manager.cc:359] Metadata directory not provided
I20260808 04:08:58.588915 2419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:08:58.589192 2419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:08:58.591886 2419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161826605655-5452-0/minicluster-data/ts-1/data/instance:
uuid: "77c0ebc3ea614a10893c7820f5692539"
format_stamp: "Formatted at 2026-08-08 04:08:58 on dist-test-slave-1l3l"
server_key: "0dc79b5a4aa30f6c65cca470c20fab55"
server_key_iv: "a04defc1402135a7355fc798c3a68697"
server_key_version: "encryptionkey@0"
I20260808 04:08:58.592620 2419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161826605655-5452-0/minicluster-data/ts-1/wal/instance:
uuid: "77c0ebc3ea614a10893c7820f5692539"
format_stamp: "Formatted at 2026-08-08 04:08:58 on dist-test-slave-1l3l"
server_key: "0dc79b5a4aa30f6c65cca470c20fab55"
server_key_iv: "a04defc1402135a7355fc798c3a68697"
server_key_version: "encryptionkey@0"
I20260808 04:08:58.597289 2419 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 04:08:58.600428 2437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:08:58.601626 2419 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:08:58.601748 2419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161826605655-5452-0/minicluster-data/ts-1/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161826605655-5452-0/minicluster-data/ts-1/wal
uuid: "77c0ebc3ea614a10893c7820f5692539"
format_stamp: "Formatted at 2026-08-08 04:08:58 on dist-test-slave-1l3l"
server_key: "0dc79b5a4aa30f6c65cca470c20fab55"
server_key_iv: "a04defc1402135a7355fc798c3a68697"
server_key_version: "encryptionkey@0"
I20260808 04:08:58.601912 2419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161826605655-5452-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161826605655-5452-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161826605655-5452-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:08:58.626466 2419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:08:58.627169 2419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:08:58.627408 2419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:08:58.628345 2419 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:08:58.629771 2419 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:08:58.629863 2419 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:08:58.629940 2419 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:08:58.629997 2419 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:08:58.660151 2419 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.2:42147
I20260808 04:08:58.660213 2588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.2:42147 every 8 connection(s)
I20260808 04:08:58.661427 2419 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161826605655-5452-0/minicluster-data/ts-1/data/info.pb
I20260808 04:08:58.662042 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 2419
I20260808 04:08:58.662194 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161826605655-5452-0/minicluster-data/ts-1/wal/instance
I20260808 04:08:58.662711 5452 external_mini_cluster.cc:1428] Setting key 27edb170608925464fe68e5ae825817f
I20260808 04:08:58.666095 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161826605655-5452-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161826605655-5452-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161826605655-5452-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.3:0
--local_ip_for_outbound_sockets=127.5.83.3
--webserver_interface=127.5.83.3
--webserver_port=0
--tserver_master_addrs=127.5.83.62:37889
--builtin_ntp_servers=127.5.83.20:35447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 04:08:58.674654 2417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:58.664364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.2:33555 (local address 127.5.83.62:37889)
0808 04:08:58.664548 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:58.664554 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:58.666217 (+ 1663us) server_negotiation.cc:407] Connection header received
0808 04:08:58.668109 (+ 1892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:58.668114 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:58.668194 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:58.668282 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:08:58.669373 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:58.670081 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:58.671589 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:58.672243 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:58.672367 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:58.673867 (+ 1500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:08:58.673873 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:08:58.673878 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:08:58.673917 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:08:58.673955 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:08:58.673970 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:08:58.674055 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:58.674268 (+ 213us) server_negotiation.cc:299] Negotiation successful
0808 04:08:58.674357 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 04:08:58.675828 2591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:58.665113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:37889 (local address 127.5.83.2:33555)
0808 04:08:58.666068 (+ 955us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:58.666112 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 04:08:58.667863 (+ 1751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:08:58.668519 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:08:58.668531 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:08:58.668637 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:08:58.669212 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:58.669229 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:58.670236 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:58.670241 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:58.671451 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:58.671464 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:58.672495 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:58.672502 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:58.672767 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:58.673536 (+ 769us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:08:58.673540 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:08:58.673587 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:08:58.673681 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:08:58.674069 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:08:58.674080 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:08:58.674087 (+ 7us) client_negotiation.cc:769] Sending connection context
0808 04:08:58.674255 (+ 168us) client_negotiation.cc:240] Negotiation successful
0808 04:08:58.674622 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":333,"threads_started":1}
I20260808 04:08:58.677158 2589 heartbeater.cc:347] Connected to a master server at 127.5.83.62:37889
I20260808 04:08:58.677450 2589 heartbeater.cc:464] Registering TS with master...
I20260808 04:08:58.678251 2589 heartbeater.cc:511] Master 127.5.83.62:37889 requested a full tablet report, sending...
I20260808 04:08:58.679735 2147 ts_manager.cc:194] Registered new tserver with Master: 77c0ebc3ea614a10893c7820f5692539 (127.5.83.2:42147)
I20260808 04:08:58.680523 2147 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.83.2:33555
W20260808 04:08:58.991142 2592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:08:58.991401 2592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:08:58.991524 2592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:08:58.998443 2592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:08:58.998528 2592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:08:58.998658 2592 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.3
I20260808 04:08:59.007421 2592 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:35447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161826605655-5452-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161826605655-5452-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.83.3
--webserver_port=0
--tserver_master_addrs=127.5.83.62:37889
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.3
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161826605655-5452-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:08:59.008565 2592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:08:59.010200 2592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:08:59.022486 2600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:08:59.022578 2601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:08:59.022691 2603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:08:59.023329 2592 server_base.cc:1034] running on GCE node
I20260808 04:08:59.023927 2592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:08:59.024847 2592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:08:59.026041 2592 hybrid_clock.cc:648] HybridClock initialized: now 1786162139025989 us; error 49 us; skew 500 ppm
I20260808 04:08:59.026320 2592 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 04:08:59.028977 2592 webserver.cc:460] Webserver started at http://127.5.83.3:33821/ using document root <none> and password file <none>
I20260808 04:08:59.029844 2592 fs_manager.cc:359] Metadata directory not provided
I20260808 04:08:59.029966 2592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:08:59.030318 2592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:08:59.033490 2592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161826605655-5452-0/minicluster-data/ts-2/data/instance:
uuid: "e00e61a093b74ae0a1d6a6807360d34c"
format_stamp: "Formatted at 2026-08-08 04:08:59 on dist-test-slave-1l3l"
server_key: "11fb0d40e679f441711b132b5ed1da0c"
server_key_iv: "f9aed36ec59c6675ab9b3df4e7471af8"
server_key_version: "encryptionkey@0"
I20260808 04:08:59.034226 2592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161826605655-5452-0/minicluster-data/ts-2/wal/instance:
uuid: "e00e61a093b74ae0a1d6a6807360d34c"
format_stamp: "Formatted at 2026-08-08 04:08:59 on dist-test-slave-1l3l"
server_key: "11fb0d40e679f441711b132b5ed1da0c"
server_key_iv: "f9aed36ec59c6675ab9b3df4e7471af8"
server_key_version: "encryptionkey@0"
I20260808 04:08:59.038990 2592 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260808 04:08:59.042150 2612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:08:59.043329 2592 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:08:59.043481 2592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161826605655-5452-0/minicluster-data/ts-2/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161826605655-5452-0/minicluster-data/ts-2/wal
uuid: "e00e61a093b74ae0a1d6a6807360d34c"
format_stamp: "Formatted at 2026-08-08 04:08:59 on dist-test-slave-1l3l"
server_key: "11fb0d40e679f441711b132b5ed1da0c"
server_key_iv: "f9aed36ec59c6675ab9b3df4e7471af8"
server_key_version: "encryptionkey@0"
I20260808 04:08:59.043668 2592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161826605655-5452-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161826605655-5452-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161826605655-5452-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:08:59.076402 2592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:08:59.077155 2592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:08:59.077414 2592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:08:59.078329 2592 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:08:59.079758 2592 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:08:59.079856 2592 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:08:59.079962 2592 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:08:59.080034 2592 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:08:59.109619 2592 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.3:36193
I20260808 04:08:59.109687 2765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.3:36193 every 8 connection(s)
I20260808 04:08:59.110811 2592 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161826605655-5452-0/minicluster-data/ts-2/data/info.pb
I20260808 04:08:59.117089 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 2592
I20260808 04:08:59.117259 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161826605655-5452-0/minicluster-data/ts-2/wal/instance
I20260808 04:08:59.117826 5452 external_mini_cluster.cc:1428] Setting key 3bd1276acc53de6b5b31390174fbf026
I20260808 04:08:59.123208 2417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:59.113398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.3:41881 (local address 127.5.83.62:37889)
0808 04:08:59.113600 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:59.113605 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:59.114953 (+ 1348us) server_negotiation.cc:407] Connection header received
0808 04:08:59.116671 (+ 1718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:59.116675 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:59.116742 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:59.116827 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:08:59.118156 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:59.119039 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:59.120963 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:59.121365 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:59.121464 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:59.122633 (+ 1169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:08:59.122636 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:08:59.122640 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:08:59.122656 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:08:59.122683 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:08:59.122693 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:08:59.122756 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:59.122915 (+ 159us) server_negotiation.cc:299] Negotiation successful
0808 04:08:59.122983 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 04:08:59.124177 2771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:59.113921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:37889 (local address 127.5.83.3:41881)
0808 04:08:59.114804 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:59.114840 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0808 04:08:59.116452 (+ 1612us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:08:59.117104 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:08:59.117116 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:08:59.117225 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:08:59.117992 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:59.118018 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:59.119194 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:59.119199 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:59.120836 (+ 1637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:59.120849 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:59.121543 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:59.121548 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:59.121733 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:59.122367 (+ 634us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:08:59.122370 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:08:59.122408 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:08:59.122482 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:08:59.122760 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:08:59.122766 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:08:59.122771 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 04:08:59.122891 (+ 120us) client_negotiation.cc:240] Negotiation successful
0808 04:08:59.123167 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":267,"threads_started":1}
I20260808 04:08:59.125305 2766 heartbeater.cc:347] Connected to a master server at 127.5.83.62:37889
I20260808 04:08:59.125562 2766 heartbeater.cc:464] Registering TS with master...
I20260808 04:08:59.126235 2766 heartbeater.cc:511] Master 127.5.83.62:37889 requested a full tablet report, sending...
I20260808 04:08:59.127496 2147 ts_manager.cc:194] Registered new tserver with Master: e00e61a093b74ae0a1d6a6807360d34c (127.5.83.3:36193)
I20260808 04:08:59.128314 2147 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.83.3:41881
I20260808 04:08:59.132606 5452 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:08:59.142421 2417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:59.136515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60988 (local address 127.5.83.62:37889)
0808 04:08:59.136663 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:59.136669 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:59.137103 (+ 434us) server_negotiation.cc:407] Connection header received
0808 04:08:59.137256 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:59.137262 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:59.137335 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:59.137425 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:08:59.137916 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:59.138756 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:59.140207 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:59.140609 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:59.140718 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:59.141757 (+ 1039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:08:59.141762 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:08:59.141765 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:08:59.141781 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:08:59.141811 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:08:59.141822 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:08:59.141897 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:59.142054 (+ 157us) server_negotiation.cc:299] Negotiation successful
0808 04:08:59.142141 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 04:08:59.145648 2147 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60988:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:08:59.152424 2147 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:08:59.173255 2789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:59.163272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51574 (local address 127.5.83.2:42147)
0808 04:08:59.166011 (+ 2739us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:59.166021 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:59.166055 (+ 34us) server_negotiation.cc:407] Connection header received
0808 04:08:59.166159 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:59.166167 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:59.166410 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:59.166612 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:08:59.167258 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:59.168664 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:59.170538 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:59.171998 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:59.172384 (+ 386us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:59.172659 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:59.172736 (+ 77us) server_negotiation.cc:299] Negotiation successful
0808 04:08:59.172894 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":249,"threads_started":1}
I20260808 04:08:59.173276 2788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:59.163117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.2:42147 (local address 127.0.0.1:51574)
0808 04:08:59.164249 (+ 1132us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:59.164290 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 04:08:59.164471 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:08:59.166676 (+ 2205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:08:59.166684 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:08:59.166719 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:08:59.167079 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:59.167090 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:59.168819 (+ 1729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:59.168824 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:59.170370 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:59.170382 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:59.172219 (+ 1837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:59.172225 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:59.172529 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:59.172589 (+ 60us) client_negotiation.cc:769] Sending connection context
0808 04:08:59.172703 (+ 114us) client_negotiation.cc:240] Negotiation successful
0808 04:08:59.172902 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":868,"mutex_wait_us":38,"thread_start_us":717,"threads_started":1}
I20260808 04:08:59.175248 2790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:59.164627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.3:36193 (local address 127.0.0.1:59654)
0808 04:08:59.168474 (+ 3847us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:59.168490 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:08:59.168597 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:08:59.169265 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:08:59.169269 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:08:59.169300 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:08:59.169684 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:59.169692 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:59.170923 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:59.170927 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:59.172858 (+ 1931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:59.172871 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:59.174329 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:59.174336 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:59.174510 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:59.174720 (+ 210us) client_negotiation.cc:769] Sending connection context
0808 04:08:59.174807 (+ 87us) client_negotiation.cc:240] Negotiation successful
0808 04:08:59.174935 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3733,"mutex_wait_us":25,"thread_start_us":882,"threads_started":1}
I20260808 04:08:59.176052 2799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:59.165232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59654 (local address 127.5.83.3:36193)
0808 04:08:59.165985 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:59.166005 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:59.168709 (+ 2704us) server_negotiation.cc:407] Connection header received
0808 04:08:59.168806 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:59.168814 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:59.169095 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:59.169251 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:08:59.169841 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:59.170792 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:59.173075 (+ 2283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:59.174153 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:59.174270 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:59.174541 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:59.175516 (+ 975us) server_negotiation.cc:299] Negotiation successful
0808 04:08:59.175675 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":402,"threads_started":1}
I20260808 04:08:59.177794 2797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:59.163418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:41573 (local address 127.0.0.1:58392)
0808 04:08:59.165174 (+ 1756us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:59.165192 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 04:08:59.165297 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:08:59.167814 (+ 2517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:08:59.167819 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:08:59.167844 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:08:59.168294 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:59.168308 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:59.171986 (+ 3678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:59.171991 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:59.174132 (+ 2141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:59.174146 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:59.175953 (+ 1807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:59.175959 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:59.176123 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:59.176154 (+ 31us) client_negotiation.cc:769] Sending connection context
0808 04:08:59.176219 (+ 65us) client_negotiation.cc:240] Negotiation successful
0808 04:08:59.177498 (+ 1279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1633,"mutex_wait_us":4,"thread_start_us":4787,"threads_started":1}
I20260808 04:08:59.177979 2791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:59.163927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58392 (local address 127.5.83.1:41573)
0808 04:08:59.167246 (+ 3319us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:59.167254 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:59.167284 (+ 30us) server_negotiation.cc:407] Connection header received
0808 04:08:59.167379 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:59.167386 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:59.167602 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:59.167823 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:08:59.168518 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:59.171825 (+ 3307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:59.174361 (+ 2536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:59.175753 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:59.177038 (+ 1285us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:59.177417 (+ 379us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:59.177518 (+ 101us) server_negotiation.cc:299] Negotiation successful
0808 04:08:59.177677 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3155,"thread_start_us":3050,"threads_started":1}
I20260808 04:08:59.177512 2507 tablet_service.cc:1467] Processing CreateTablet for tablet 0a109a5fb01d4d51b3592ed555d2894c (DEFAULT_TABLE table=test-table [id=79224b9288f94b93a7670f986c4561f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:08:59.179325 2507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a109a5fb01d4d51b3592ed555d2894c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:08:59.179451 2677 tablet_service.cc:1467] Processing CreateTablet for tablet 0a109a5fb01d4d51b3592ed555d2894c (DEFAULT_TABLE table=test-table [id=79224b9288f94b93a7670f986c4561f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:08:59.180841 2677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a109a5fb01d4d51b3592ed555d2894c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:08:59.181447 2326 tablet_service.cc:1467] Processing CreateTablet for tablet 0a109a5fb01d4d51b3592ed555d2894c (DEFAULT_TABLE table=test-table [id=79224b9288f94b93a7670f986c4561f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:08:59.182710 2326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a109a5fb01d4d51b3592ed555d2894c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:08:59.190732 2800 tablet_bootstrap.cc:492] T 0a109a5fb01d4d51b3592ed555d2894c P 77c0ebc3ea614a10893c7820f5692539: Bootstrap starting.
I20260808 04:08:59.193549 2800 tablet_bootstrap.cc:654] T 0a109a5fb01d4d51b3592ed555d2894c P 77c0ebc3ea614a10893c7820f5692539: Neither blocks nor log segments found. Creating new log.
I20260808 04:08:59.194373 2802 tablet_bootstrap.cc:492] T 0a109a5fb01d4d51b3592ed555d2894c P e00e61a093b74ae0a1d6a6807360d34c: Bootstrap starting.
I20260808 04:08:59.194800 2800 log.cc:824] T 0a109a5fb01d4d51b3592ed555d2894c P 77c0ebc3ea614a10893c7820f5692539: Log is configured to *not* fsync() on all Append() calls
I20260808 04:08:59.196936 2807 tablet_bootstrap.cc:492] T 0a109a5fb01d4d51b3592ed555d2894c P 0efad64a49314c748e6e42fe4c19d2dd: Bootstrap starting.
I20260808 04:08:59.197682 2802 tablet_bootstrap.cc:654] T 0a109a5fb01d4d51b3592ed555d2894c P e00e61a093b74ae0a1d6a6807360d34c: Neither blocks nor log segments found. Creating new log.
I20260808 04:08:59.197973 2800 tablet_bootstrap.cc:492] T 0a109a5fb01d4d51b3592ed555d2894c P 77c0ebc3ea614a10893c7820f5692539: No bootstrap required, opened a new log
I20260808 04:08:59.198238 2800 ts_tablet_manager.cc:1397] T 0a109a5fb01d4d51b3592ed555d2894c P 77c0ebc3ea614a10893c7820f5692539: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260808 04:08:59.198621 2802 log.cc:824] T 0a109a5fb01d4d51b3592ed555d2894c P e00e61a093b74ae0a1d6a6807360d34c: Log is configured to *not* fsync() on all Append() calls
I20260808 04:08:59.200187 2807 tablet_bootstrap.cc:654] T 0a109a5fb01d4d51b3592ed555d2894c P 0efad64a49314c748e6e42fe4c19d2dd: Neither blocks nor log segments found. Creating new log.
I20260808 04:08:59.200896 2802 tablet_bootstrap.cc:492] T 0a109a5fb01d4d51b3592ed555d2894c P e00e61a093b74ae0a1d6a6807360d34c: No bootstrap required, opened a new log
I20260808 04:08:59.201110 2802 ts_tablet_manager.cc:1397] T 0a109a5fb01d4d51b3592ed555d2894c P e00e61a093b74ae0a1d6a6807360d34c: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260808 04:08:59.201270 2807 log.cc:824] T 0a109a5fb01d4d51b3592ed555d2894c P 0efad64a49314c748e6e42fe4c19d2dd: Log is configured to *not* fsync() on all Append() calls
I20260808 04:08:59.204125 2807 tablet_bootstrap.cc:492] T 0a109a5fb01d4d51b3592ed555d2894c P 0efad64a49314c748e6e42fe4c19d2dd: No bootstrap required, opened a new log
I20260808 04:08:59.204355 2807 ts_tablet_manager.cc:1397] T 0a109a5fb01d4d51b3592ed555d2894c P 0efad64a49314c748e6e42fe4c19d2dd: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260808 04:08:59.207700 2800 raft_consensus.cc:348] T 0a109a5fb01d4d51b3592ed555d2894c P 77c0ebc3ea614a10893c7820f5692539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c0ebc3ea614a10893c7820f5692539" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 42147 } } peers { permanent_uuid: "0efad64a49314c748e6e42fe4c19d2dd" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 41573 } } peers { permanent_uuid: "e00e61a093b74ae0a1d6a6807360d34c" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 36193 } }
I20260808 04:08:59.208178 2800 raft_consensus.cc:374] T 0a109a5fb01d4d51b3592ed555d2894c P 77c0ebc3ea614a10893c7820f5692539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:08:59.208278 2800 raft_consensus.cc:729] T 0a109a5fb01d4d51b3592ed555d2894c P 77c0ebc3ea614a10893c7820f5692539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77c0ebc3ea614a10893c7820f5692539, State: Initialized, Role: FOLLOWER
I20260808 04:08:59.208596 2802 raft_consensus.cc:348] T 0a109a5fb01d4d51b3592ed555d2894c P e00e61a093b74ae0a1d6a6807360d34c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c0ebc3ea614a10893c7820f5692539" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 42147 } } peers { permanent_uuid: "0efad64a49314c748e6e42fe4c19d2dd" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 41573 } } peers { permanent_uuid: "e00e61a093b74ae0a1d6a6807360d34c" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 36193 } }
I20260808 04:08:59.208918 2802 raft_consensus.cc:374] T 0a109a5fb01d4d51b3592ed555d2894c P e00e61a093b74ae0a1d6a6807360d34c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:08:59.209012 2802 raft_consensus.cc:729] T 0a109a5fb01d4d51b3592ed555d2894c P e00e61a093b74ae0a1d6a6807360d34c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e00e61a093b74ae0a1d6a6807360d34c, State: Initialized, Role: FOLLOWER
I20260808 04:08:59.208856 2800 consensus_queue.cc:260] T 0a109a5fb01d4d51b3592ed555d2894c P 77c0ebc3ea614a10893c7820f5692539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c0ebc3ea614a10893c7820f5692539" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 42147 } } peers { permanent_uuid: "0efad64a49314c748e6e42fe4c19d2dd" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 41573 } } peers { permanent_uuid: "e00e61a093b74ae0a1d6a6807360d34c" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 36193 } }
I20260808 04:08:59.209504 2802 consensus_queue.cc:260] T 0a109a5fb01d4d51b3592ed555d2894c P e00e61a093b74ae0a1d6a6807360d34c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c0ebc3ea614a10893c7820f5692539" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 42147 } } peers { permanent_uuid: "0efad64a49314c748e6e42fe4c19d2dd" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 41573 } } peers { permanent_uuid: "e00e61a093b74ae0a1d6a6807360d34c" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 36193 } }
I20260808 04:08:59.210453 2766 heartbeater.cc:503] Master 127.5.83.62:37889 was elected leader, sending a full tablet report...
I20260808 04:08:59.211938 2589 heartbeater.cc:503] Master 127.5.83.62:37889 was elected leader, sending a full tablet report...
I20260808 04:08:59.212322 2800 ts_tablet_manager.cc:1428] T 0a109a5fb01d4d51b3592ed555d2894c P 77c0ebc3ea614a10893c7820f5692539: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260808 04:08:59.213757 2802 ts_tablet_manager.cc:1428] T 0a109a5fb01d4d51b3592ed555d2894c P e00e61a093b74ae0a1d6a6807360d34c: Time spent starting tablet: real 0.013s user 0.004s sys 0.007s
I20260808 04:08:59.213860 2807 raft_consensus.cc:348] T 0a109a5fb01d4d51b3592ed555d2894c P 0efad64a49314c748e6e42fe4c19d2dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c0ebc3ea614a10893c7820f5692539" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 42147 } } peers { permanent_uuid: "0efad64a49314c748e6e42fe4c19d2dd" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 41573 } } peers { permanent_uuid: "e00e61a093b74ae0a1d6a6807360d34c" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 36193 } }
I20260808 04:08:59.214258 2807 raft_consensus.cc:374] T 0a109a5fb01d4d51b3592ed555d2894c P 0efad64a49314c748e6e42fe4c19d2dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:08:59.214354 2807 raft_consensus.cc:729] T 0a109a5fb01d4d51b3592ed555d2894c P 0efad64a49314c748e6e42fe4c19d2dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0efad64a49314c748e6e42fe4c19d2dd, State: Initialized, Role: FOLLOWER
I20260808 04:08:59.214910 2807 consensus_queue.cc:260] T 0a109a5fb01d4d51b3592ed555d2894c P 0efad64a49314c748e6e42fe4c19d2dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c0ebc3ea614a10893c7820f5692539" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 42147 } } peers { permanent_uuid: "0efad64a49314c748e6e42fe4c19d2dd" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 41573 } } peers { permanent_uuid: "e00e61a093b74ae0a1d6a6807360d34c" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 36193 } }
I20260808 04:08:59.218194 2813 raft_consensus.cc:482] T 0a109a5fb01d4d51b3592ed555d2894c P 77c0ebc3ea614a10893c7820f5692539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:08:59.218456 2813 raft_consensus.cc:504] T 0a109a5fb01d4d51b3592ed555d2894c P 77c0ebc3ea614a10893c7820f5692539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c0ebc3ea614a10893c7820f5692539" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 42147 } } peers { permanent_uuid: "0efad64a49314c748e6e42fe4c19d2dd" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 41573 } } peers { permanent_uuid: "e00e61a093b74ae0a1d6a6807360d34c" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 36193 } }
I20260808 04:08:59.220862 2807 ts_tablet_manager.cc:1428] T 0a109a5fb01d4d51b3592ed555d2894c P 0efad64a49314c748e6e42fe4c19d2dd: Time spent starting tablet: real 0.016s user 0.011s sys 0.005s
I20260808 04:08:59.221477 2415 heartbeater.cc:503] Master 127.5.83.62:37889 was elected leader, sending a full tablet report...
I20260808 04:08:59.224210 2813 leader_election.cc:288] T 0a109a5fb01d4d51b3592ed555d2894c P 77c0ebc3ea614a10893c7820f5692539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0efad64a49314c748e6e42fe4c19d2dd (127.5.83.1:41573), e00e61a093b74ae0a1d6a6807360d34c (127.5.83.3:36193)
I20260808 04:08:59.231688 2817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:59.220261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.3:36193 (local address 127.5.83.2:54651)
0808 04:08:59.224864 (+ 4603us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:59.224884 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 04:08:59.225069 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:08:59.225546 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:08:59.225551 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:08:59.225576 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:08:59.225907 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:59.225918 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:59.227226 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:59.227230 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:59.229245 (+ 2015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:59.229260 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:59.231018 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:59.231025 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:59.231202 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:59.231235 (+ 33us) client_negotiation.cc:769] Sending connection context
0808 04:08:59.231330 (+ 95us) client_negotiation.cc:240] Negotiation successful
0808 04:08:59.231453 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4450,"thread_start_us":4266,"threads_started":1}
I20260808 04:08:59.231753 2799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:59.220752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.2:54651 (local address 127.5.83.3:36193)
0808 04:08:59.221122 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:59.221128 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:59.224963 (+ 3835us) server_negotiation.cc:407] Connection header received
0808 04:08:59.225230 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:59.225236 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:59.225308 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:59.225362 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:08:59.226049 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:59.227090 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:59.229429 (+ 2339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:59.230657 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:59.230776 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:59.230881 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:59.231361 (+ 480us) server_negotiation.cc:299] Negotiation successful
0808 04:08:59.231477 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247}
I20260808 04:08:59.232687 2705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a109a5fb01d4d51b3592ed555d2894c" candidate_uuid: "77c0ebc3ea614a10893c7820f5692539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e00e61a093b74ae0a1d6a6807360d34c" is_pre_election: true
I20260808 04:08:59.233165 2818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:59.220329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:41573 (local address 127.5.83.2:56763)
0808 04:08:59.227284 (+ 6955us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:59.227299 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:08:59.227416 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:08:59.227741 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:08:59.227745 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:08:59.227774 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:08:59.228125 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:59.228133 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:59.229425 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:59.229431 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:59.231362 (+ 1931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:08:59.231374 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:08:59.232691 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:08:59.232695 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:08:59.232810 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:59.232832 (+ 22us) client_negotiation.cc:769] Sending connection context
0808 04:08:59.232902 (+ 70us) client_negotiation.cc:240] Negotiation successful
0808 04:08:59.232978 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6827,"thread_start_us":6666,"threads_started":1}
I20260808 04:08:59.233186 2705 raft_consensus.cc:2454] T 0a109a5fb01d4d51b3592ed555d2894c P e00e61a093b74ae0a1d6a6807360d34c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77c0ebc3ea614a10893c7820f5692539 in term 0.
I20260808 04:08:59.233302 2791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:59.220932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.2:56763 (local address 127.5.83.1:41573)
0808 04:08:59.221112 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:59.221118 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:59.227403 (+ 6285us) server_negotiation.cc:407] Connection header received
0808 04:08:59.227520 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:59.227525 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:59.227604 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:59.227695 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:08:59.228276 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:59.229269 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:59.231545 (+ 2276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:59.232517 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:59.232640 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:59.232724 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:59.232950 (+ 226us) server_negotiation.cc:299] Negotiation successful
0808 04:08:59.233068 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260808 04:08:59.233943 2442 leader_election.cc:302] T 0a109a5fb01d4d51b3592ed555d2894c P 77c0ebc3ea614a10893c7820f5692539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77c0ebc3ea614a10893c7820f5692539, e00e61a093b74ae0a1d6a6807360d34c; no voters:
I20260808 04:08:59.234127 2353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a109a5fb01d4d51b3592ed555d2894c" candidate_uuid: "77c0ebc3ea614a10893c7820f5692539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0efad64a49314c748e6e42fe4c19d2dd" is_pre_election: true
I20260808 04:08:59.234368 2813 raft_consensus.cc:2781] T 0a109a5fb01d4d51b3592ed555d2894c P 77c0ebc3ea614a10893c7820f5692539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:08:59.234515 2813 raft_consensus.cc:482] T 0a109a5fb01d4d51b3592ed555d2894c P 77c0ebc3ea614a10893c7820f5692539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:08:59.234613 2813 raft_consensus.cc:3037] T 0a109a5fb01d4d51b3592ed555d2894c P 77c0ebc3ea614a10893c7820f5692539 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:08:59.234606 2353 raft_consensus.cc:2454] T 0a109a5fb01d4d51b3592ed555d2894c P 0efad64a49314c748e6e42fe4c19d2dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77c0ebc3ea614a10893c7820f5692539 in term 0.
I20260808 04:08:59.236994 2813 raft_consensus.cc:504] T 0a109a5fb01d4d51b3592ed555d2894c P 77c0ebc3ea614a10893c7820f5692539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c0ebc3ea614a10893c7820f5692539" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 42147 } } peers { permanent_uuid: "0efad64a49314c748e6e42fe4c19d2dd" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 41573 } } peers { permanent_uuid: "e00e61a093b74ae0a1d6a6807360d34c" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 36193 } }
I20260808 04:08:59.237495 2813 leader_election.cc:288] T 0a109a5fb01d4d51b3592ed555d2894c P 77c0ebc3ea614a10893c7820f5692539 [CANDIDATE]: Term 1 election: Requested vote from peers 0efad64a49314c748e6e42fe4c19d2dd (127.5.83.1:41573), e00e61a093b74ae0a1d6a6807360d34c (127.5.83.3:36193)
I20260808 04:08:59.237870 2353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a109a5fb01d4d51b3592ed555d2894c" candidate_uuid: "77c0ebc3ea614a10893c7820f5692539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0efad64a49314c748e6e42fe4c19d2dd"
I20260808 04:08:59.237912 2705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a109a5fb01d4d51b3592ed555d2894c" candidate_uuid: "77c0ebc3ea614a10893c7820f5692539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e00e61a093b74ae0a1d6a6807360d34c"
I20260808 04:08:59.238032 2353 raft_consensus.cc:3037] T 0a109a5fb01d4d51b3592ed555d2894c P 0efad64a49314c748e6e42fe4c19d2dd [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:08:59.238057 2705 raft_consensus.cc:3037] T 0a109a5fb01d4d51b3592ed555d2894c P e00e61a093b74ae0a1d6a6807360d34c [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:08:59.240159 2353 raft_consensus.cc:2454] T 0a109a5fb01d4d51b3592ed555d2894c P 0efad64a49314c748e6e42fe4c19d2dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77c0ebc3ea614a10893c7820f5692539 in term 1.
I20260808 04:08:59.240162 2705 raft_consensus.cc:2454] T 0a109a5fb01d4d51b3592ed555d2894c P e00e61a093b74ae0a1d6a6807360d34c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77c0ebc3ea614a10893c7820f5692539 in term 1.
I20260808 04:08:59.240500 2442 leader_election.cc:302] T 0a109a5fb01d4d51b3592ed555d2894c P 77c0ebc3ea614a10893c7820f5692539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77c0ebc3ea614a10893c7820f5692539, e00e61a093b74ae0a1d6a6807360d34c; no voters:
I20260808 04:08:59.240741 2813 raft_consensus.cc:2781] T 0a109a5fb01d4d51b3592ed555d2894c P 77c0ebc3ea614a10893c7820f5692539 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:08:59.241148 2813 raft_consensus.cc:686] T 0a109a5fb01d4d51b3592ed555d2894c P 77c0ebc3ea614a10893c7820f5692539 [term 1 LEADER]: Becoming Leader. State: Replica: 77c0ebc3ea614a10893c7820f5692539, State: Running, Role: LEADER
I20260808 04:08:59.241549 2813 consensus_queue.cc:237] T 0a109a5fb01d4d51b3592ed555d2894c P 77c0ebc3ea614a10893c7820f5692539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c0ebc3ea614a10893c7820f5692539" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 42147 } } peers { permanent_uuid: "0efad64a49314c748e6e42fe4c19d2dd" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 41573 } } peers { permanent_uuid: "e00e61a093b74ae0a1d6a6807360d34c" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 36193 } }
I20260808 04:08:59.245854 2147 catalog_manager.cc:5495] T 0a109a5fb01d4d51b3592ed555d2894c P 77c0ebc3ea614a10893c7820f5692539 reported cstate change: term changed from 0 to 1, leader changed from <none> to 77c0ebc3ea614a10893c7820f5692539 (127.5.83.2). New cstate: current_term: 1 leader_uuid: "77c0ebc3ea614a10893c7820f5692539" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c0ebc3ea614a10893c7820f5692539" member_type: VOTER last_known_addr { host: "127.5.83.2" port: 42147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0efad64a49314c748e6e42fe4c19d2dd" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 41573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e00e61a093b74ae0a1d6a6807360d34c" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 36193 } health_report { overall_health: UNKNOWN } } }
I20260808 04:08:59.277737 2789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:59.272766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51578 (local address 127.5.83.2:42147)
0808 04:08:59.272976 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:59.272980 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:59.272993 (+ 13us) server_negotiation.cc:407] Connection header received
0808 04:08:59.273064 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:59.273067 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:59.273144 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:59.273270 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:08:59.273641 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:59.274353 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:59.275730 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:59.276154 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:59.276264 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:59.276732 (+ 468us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:08:59.277197 (+ 465us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:08:59.277323 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:59.277434 (+ 111us) server_negotiation.cc:299] Negotiation successful
0808 04:08:59.277510 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 04:08:59.285049 2705 raft_consensus.cc:1261] T 0a109a5fb01d4d51b3592ed555d2894c P e00e61a093b74ae0a1d6a6807360d34c [term 1 FOLLOWER]: Refusing update from remote peer 77c0ebc3ea614a10893c7820f5692539: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:08:59.285055 2353 raft_consensus.cc:1261] T 0a109a5fb01d4d51b3592ed555d2894c P 0efad64a49314c748e6e42fe4c19d2dd [term 1 FOLLOWER]: Refusing update from remote peer 77c0ebc3ea614a10893c7820f5692539: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:08:59.285894 2819 consensus_queue.cc:1035] T 0a109a5fb01d4d51b3592ed555d2894c P 77c0ebc3ea614a10893c7820f5692539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0efad64a49314c748e6e42fe4c19d2dd" member_type: VOTER last_known_addr { host: "127.5.83.1" port: 41573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:08:59.286180 2813 consensus_queue.cc:1035] T 0a109a5fb01d4d51b3592ed555d2894c P 77c0ebc3ea614a10893c7820f5692539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e00e61a093b74ae0a1d6a6807360d34c" member_type: VOTER last_known_addr { host: "127.5.83.3" port: 36193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:08:59.307754 2146 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:08:59.308068 2146 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:08:59.311599 2146 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eb9e4100dee94a3687369cfbbafacece: Sending DeleteTablet for 3 replicas of tablet 0a109a5fb01d4d51b3592ed555d2894c
I20260808 04:08:59.312685 2507 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a109a5fb01d4d51b3592ed555d2894c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:08:59 UTC) from {username='slave'} at 127.0.0.1:51574
I20260808 04:08:59.312765 2326 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a109a5fb01d4d51b3592ed555d2894c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:08:59 UTC) from {username='slave'} at 127.0.0.1:58392
I20260808 04:08:59.314002 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 2238
I20260808 04:08:59.313941 2677 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a109a5fb01d4d51b3592ed555d2894c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:08:59 UTC) from {username='slave'} at 127.0.0.1:59654
I20260808 04:08:59.333227 2791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:59.315502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58402 (local address 127.5.83.1:41573)
0808 04:08:59.315670 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:59.315676 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:59.325193 (+ 9517us) server_negotiation.cc:407] Connection header received
0808 04:08:59.325370 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:59.325376 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:59.325477 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:59.325588 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:08:59.326047 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:59.327252 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:59.329181 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:59.329863 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:59.329955 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:59.331449 (+ 1494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:08:59.331461 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:08:59.331480 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:08:59.332292 (+ 812us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:08:59.332351 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:08:59.332377 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:08:59.332470 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:59.332715 (+ 245us) server_negotiation.cc:299] Negotiation successful
0808 04:08:59.332843 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260808 04:08:59.334631 2407 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:08:59.340194 2835 tablet_replica.cc:330] T 0a109a5fb01d4d51b3592ed555d2894c P 77c0ebc3ea614a10893c7820f5692539: stopping tablet replica
I20260808 04:08:59.340571 2835 raft_consensus.cc:2229] T 0a109a5fb01d4d51b3592ed555d2894c P 77c0ebc3ea614a10893c7820f5692539 [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:08:59.341135 2835 raft_consensus.cc:2258] T 0a109a5fb01d4d51b3592ed555d2894c P 77c0ebc3ea614a10893c7820f5692539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:08:59.341881 2841 tablet_replica.cc:330] T 0a109a5fb01d4d51b3592ed555d2894c P e00e61a093b74ae0a1d6a6807360d34c: stopping tablet replica
I20260808 04:08:59.342340 2841 raft_consensus.cc:2229] T 0a109a5fb01d4d51b3592ed555d2894c P e00e61a093b74ae0a1d6a6807360d34c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:08:59.342891 2841 raft_consensus.cc:2258] T 0a109a5fb01d4d51b3592ed555d2894c P e00e61a093b74ae0a1d6a6807360d34c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:08:59.343376 2835 ts_tablet_manager.cc:1905] T 0a109a5fb01d4d51b3592ed555d2894c P 77c0ebc3ea614a10893c7820f5692539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:08:59.345052 2841 ts_tablet_manager.cc:1905] T 0a109a5fb01d4d51b3592ed555d2894c P e00e61a093b74ae0a1d6a6807360d34c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:08:59.350931 2835 ts_tablet_manager.cc:1918] T 0a109a5fb01d4d51b3592ed555d2894c P 77c0ebc3ea614a10893c7820f5692539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:08:59.351155 2835 log.cc:1196] T 0a109a5fb01d4d51b3592ed555d2894c P 77c0ebc3ea614a10893c7820f5692539: Deleting WAL directory at /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161826605655-5452-0/minicluster-data/ts-1/wal/wals/0a109a5fb01d4d51b3592ed555d2894c
I20260808 04:08:59.351814 2835 ts_tablet_manager.cc:1939] T 0a109a5fb01d4d51b3592ed555d2894c P 77c0ebc3ea614a10893c7820f5692539: Deleting consensus metadata
I20260808 04:08:59.351881 2841 ts_tablet_manager.cc:1918] T 0a109a5fb01d4d51b3592ed555d2894c P e00e61a093b74ae0a1d6a6807360d34c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:08:59.352084 2841 log.cc:1196] T 0a109a5fb01d4d51b3592ed555d2894c P e00e61a093b74ae0a1d6a6807360d34c: Deleting WAL directory at /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161826605655-5452-0/minicluster-data/ts-2/wal/wals/0a109a5fb01d4d51b3592ed555d2894c
I20260808 04:08:59.352617 2841 ts_tablet_manager.cc:1939] T 0a109a5fb01d4d51b3592ed555d2894c P e00e61a093b74ae0a1d6a6807360d34c: Deleting consensus metadata
I20260808 04:08:59.353677 2129 catalog_manager.cc:4841] TS 77c0ebc3ea614a10893c7820f5692539 (127.5.83.2:42147): tablet 0a109a5fb01d4d51b3592ed555d2894c (table test-table [id=79224b9288f94b93a7670f986c4561f2]) successfully deleted
I20260808 04:08:59.354628 2129 catalog_manager.cc:4841] TS e00e61a093b74ae0a1d6a6807360d34c (127.5.83.3:36193): tablet 0a109a5fb01d4d51b3592ed555d2894c (table test-table [id=79224b9288f94b93a7670f986c4561f2]) successfully deleted
I20260808 04:08:59.473109 2837 tablet_replica.cc:330] T 0a109a5fb01d4d51b3592ed555d2894c P 0efad64a49314c748e6e42fe4c19d2dd: stopping tablet replica
I20260808 04:08:59.473434 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 2238
W20260808 04:08:59.494305 2128 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.83.1:41573 (error 108)
I20260808 04:08:59.494570 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 2419
I20260808 04:08:59.496448 2797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:59.495933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:41573 (local address 127.0.0.1:58410)
0808 04:08:59.496133 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:59.496264 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.1:41573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260808 04:08:59.496753 2128 catalog_manager.cc:4584] TS 0efad64a49314c748e6e42fe4c19d2dd (127.5.83.1:41573): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0a109a5fb01d4d51b3592ed555d2894c: Network error: Client connection negotiation failed: client connection to 127.5.83.1:41573: connect: Connection refused (error 111)
I20260808 04:08:59.501801 2789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:59.495433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51582 (local address 127.5.83.2:42147)
0808 04:08:59.495636 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:59.495642 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:59.495663 (+ 21us) server_negotiation.cc:407] Connection header received
0808 04:08:59.495920 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:59.495926 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:59.496040 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:59.496156 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:08:59.496576 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:59.497502 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:59.498722 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:59.499160 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:59.499270 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:59.500485 (+ 1215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:08:59.500493 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:08:59.500507 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:08:59.501085 (+ 578us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:08:59.501135 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:08:59.501155 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:08:59.501264 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:59.501393 (+ 129us) server_negotiation.cc:299] Negotiation successful
0808 04:08:59.501487 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 04:08:59.502534 2583 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:08:59.550321 2797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:59.549876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:41573 (local address 127.0.0.1:58422)
0808 04:08:59.550079 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:59.550163 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.1:41573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260808 04:08:59.551347 2797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:59.551012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:41573 (local address 127.0.0.1:58424)
0808 04:08:59.551157 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:59.551223 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.1:41573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260808 04:08:59.611256 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 2419
I20260808 04:08:59.612500 2797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:59.611363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:41573 (local address 127.0.0.1:58430)
0808 04:08:59.612179 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:59.612269 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.1:41573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":693}
I20260808 04:08:59.613554 2797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:59.613178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:41573 (local address 127.0.0.1:58432)
0808 04:08:59.613310 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:59.613387 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.1:41573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260808 04:08:59.634290 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 2592
I20260808 04:08:59.641086 2799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:59.635051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59656 (local address 127.5.83.3:36193)
0808 04:08:59.635223 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0808 04:08:59.635229 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:08:59.635251 (+ 22us) server_negotiation.cc:407] Connection header received
0808 04:08:59.635475 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:08:59.635481 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:08:59.635582 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:08:59.635667 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:08:59.636121 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:59.636843 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:59.638176 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:08:59.638563 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:08:59.638673 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:08:59.639749 (+ 1076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:08:59.639763 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:08:59.639782 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:08:59.640376 (+ 594us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:08:59.640425 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:08:59.640445 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:08:59.640554 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0808 04:08:59.640677 (+ 123us) server_negotiation.cc:299] Negotiation successful
0808 04:08:59.640784 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 04:08:59.641665 2761 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:08:59.693962 2797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:59.693463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:41573 (local address 127.0.0.1:58438)
0808 04:08:59.693686 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:59.693794 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.1:41573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260808 04:08:59.694977 2797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:08:59.694563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.1:41573 (local address 127.0.0.1:58448)
0808 04:08:59.694765 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0808 04:08:59.694841 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.1:41573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260808 04:08:59.728034 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 2592
I20260808 04:08:59.749754 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 2099
I20260808 04:08:59.750780 2190 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:08:59.857844 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 2099
2026-08-08T04:08:59Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2670 ms)
[----------] 6 tests from EncryptionPolicyTest (21259 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786161826605655-5452-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:08:59 dist-test-slave-1l3l krb5kdc[2855](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:08:59 dist-test-slave-1l3l krb5kdc[2855](info): set up 2 sockets
Aug 08 04:08:59 dist-test-slave-1l3l krb5kdc[2855](info): commencing operation
krb5kdc: starting...
W20260808 04:09:01.071223 5452 mini_kdc.cc:121] Time spent starting KDC: real 1.165s user 0.004s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786161826605655-5452-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786161826605655-5452-0/krb5kdc/test-user.keytab.
Aug 08 04:09:01 dist-test-slave-1l3l krb5kdc[2855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162141, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:09:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:09:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786161826605655-5452-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786161826605655-5452-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 04:09:01.353304 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786161826605655-5452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786161826605655-5452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786161826605655-5452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34245
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786161826605655-5452-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34245
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:09:01.676513 2873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:09:01.676765 2873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:09:01.676903 2873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:09:01.683770 2873 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:09:01.683882 2873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:09:01.683957 2873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:09:01.684037 2873 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:09:01.684096 2873 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:09:01.693476 2873 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786161826605655-5452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786161826605655-5452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34245
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786161826605655-5452-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786161826605655-5452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:09:01.694589 2873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:09:01.695827 2873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:09:01.706930 2882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:09:01.707005 2883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:09:01.706996 2886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:09:01.708074 2873 server_base.cc:1034] running on GCE node
I20260808 04:09:01.708778 2873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:09:01.709983 2873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:09:01.711186 2873 hybrid_clock.cc:648] HybridClock initialized: now 1786162141711145 us; error 28 us; skew 500 ppm
I20260808 04:09:01.711513 2873 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:09:01 dist-test-slave-1l3l krb5kdc[2855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162141, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:09:01.714418 2873 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 04:09:01.715687 2873 webserver.cc:460] Webserver started at http://127.0.0.1:41969/ using document root <none> and password file <none>
I20260808 04:09:01.716320 2873 fs_manager.cc:359] Metadata directory not provided
I20260808 04:09:01.716424 2873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:09:01.716682 2873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:09:01.719350 2873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786161826605655-5452-0/minicluster-data/master-0/data/instance:
uuid: "72027f21e48748b3be934961123b6962"
format_stamp: "Formatted at 2026-08-08 04:09:01 on dist-test-slave-1l3l"
server_key: "7ace1a4e7ebf64b47141b3bab3e2b51b"
server_key_iv: "367453a667ccc443499e3f3cbc3e5b47"
server_key_version: "encryptionkey@0"
I20260808 04:09:01.720094 2873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786161826605655-5452-0/minicluster-data/master-0/wal/instance:
uuid: "72027f21e48748b3be934961123b6962"
format_stamp: "Formatted at 2026-08-08 04:09:01 on dist-test-slave-1l3l"
server_key: "7ace1a4e7ebf64b47141b3bab3e2b51b"
server_key_iv: "367453a667ccc443499e3f3cbc3e5b47"
server_key_version: "encryptionkey@0"
I20260808 04:09:01.724769 2873 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260808 04:09:01.727960 2894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:09:01.729154 2873 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 04:09:01.729305 2873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786161826605655-5452-0/minicluster-data/master-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786161826605655-5452-0/minicluster-data/master-0/wal
uuid: "72027f21e48748b3be934961123b6962"
format_stamp: "Formatted at 2026-08-08 04:09:01 on dist-test-slave-1l3l"
server_key: "7ace1a4e7ebf64b47141b3bab3e2b51b"
server_key_iv: "367453a667ccc443499e3f3cbc3e5b47"
server_key_version: "encryptionkey@0"
I20260808 04:09:01.729493 2873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786161826605655-5452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786161826605655-5452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786161826605655-5452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:09:01.756551 2873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:09:01.760102 2873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:09:01.760352 2873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:09:01.776871 2873 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34245
I20260808 04:09:01.776904 2982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34245 every 8 connection(s)
I20260808 04:09:01.778090 2873 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
I20260808 04:09:01.781497 2985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:09:01.788055 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 2873
I20260808 04:09:01.788273 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786161826605655-5452-0/minicluster-data/master-0/wal/instance
I20260808 04:09:01.789049 5452 external_mini_cluster.cc:1428] Setting key 50e4306454954e9e5b6b999099c89f31
I20260808 04:09:01.791827 2985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72027f21e48748b3be934961123b6962: Bootstrap starting.
I20260808 04:09:01.794919 2985 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 72027f21e48748b3be934961123b6962: Neither blocks nor log segments found. Creating new log.
I20260808 04:09:01.796176 2985 log.cc:824] T 00000000000000000000000000000000 P 72027f21e48748b3be934961123b6962: Log is configured to *not* fsync() on all Append() calls
I20260808 04:09:01.799172 2985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72027f21e48748b3be934961123b6962: No bootstrap required, opened a new log
Aug 08 04:09:01 dist-test-slave-1l3l krb5kdc[2855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162141, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260808 04:09:01.807586 2985 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72027f21e48748b3be934961123b6962 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72027f21e48748b3be934961123b6962" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34245 } }
I20260808 04:09:01.807988 2985 raft_consensus.cc:374] T 00000000000000000000000000000000 P 72027f21e48748b3be934961123b6962 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:09:01.808106 2985 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72027f21e48748b3be934961123b6962 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72027f21e48748b3be934961123b6962, State: Initialized, Role: FOLLOWER
I20260808 04:09:01.808732 2985 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72027f21e48748b3be934961123b6962 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72027f21e48748b3be934961123b6962" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34245 } }
I20260808 04:09:01.808987 2985 raft_consensus.cc:388] T 00000000000000000000000000000000 P 72027f21e48748b3be934961123b6962 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:09:01.809139 2985 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72027f21e48748b3be934961123b6962 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:09:01.809327 2985 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72027f21e48748b3be934961123b6962 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:09:01.811807 2992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:01.790854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38859 (local address 127.0.0.1:34245)
0808 04:09:01.791562 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:01.791572 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:01.791612 (+ 40us) server_negotiation.cc:407] Connection header received
0808 04:09:01.793332 (+ 1720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:01.793352 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:01.793698 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:01.793982 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:01.795878 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:01.797531 (+ 1653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:01.799358 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:01.800047 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:01.800176 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:01.804785 (+ 4609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:09:01.804793 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:09:01.804808 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:09:01.804842 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:09:01.807806 (+ 2964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:09:01.808636 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:09:01.808641 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:09:01.808646 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:09:01.808807 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:09:01.809309 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:09:01.809317 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:09:01.809321 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:09:01.809853 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:09:01.810115 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:01.810555 (+ 440us) server_negotiation.cc:299] Negotiation successful
0808 04:09:01.810843 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":322,"threads_started":1}
I20260808 04:09:01.811852 2985 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72027f21e48748b3be934961123b6962 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72027f21e48748b3be934961123b6962" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34245 } }
I20260808 04:09:01.812300 2985 leader_election.cc:302] T 00000000000000000000000000000000 P 72027f21e48748b3be934961123b6962 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 72027f21e48748b3be934961123b6962; no voters:
I20260808 04:09:01.812844 2985 leader_election.cc:288] T 00000000000000000000000000000000 P 72027f21e48748b3be934961123b6962 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:09:01.812971 2994 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 72027f21e48748b3be934961123b6962 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:09:01.814024 2994 raft_consensus.cc:686] T 00000000000000000000000000000000 P 72027f21e48748b3be934961123b6962 [term 1 LEADER]: Becoming Leader. State: Replica: 72027f21e48748b3be934961123b6962, State: Running, Role: LEADER
I20260808 04:09:01.814601 2994 consensus_queue.cc:237] T 00000000000000000000000000000000 P 72027f21e48748b3be934961123b6962 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72027f21e48748b3be934961123b6962" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34245 } }
I20260808 04:09:01.814852 2985 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72027f21e48748b3be934961123b6962 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:09:01.816037 5452 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:09:01.817214 2995 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72027f21e48748b3be934961123b6962 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "72027f21e48748b3be934961123b6962" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72027f21e48748b3be934961123b6962" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34245 } } }
I20260808 04:09:01.817585 2995 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72027f21e48748b3be934961123b6962 [sys.catalog]: This master's current role is: LEADER
I20260808 04:09:01.822999 2999 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72027f21e48748b3be934961123b6962 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72027f21e48748b3be934961123b6962. Latest consensus state: current_term: 1 leader_uuid: "72027f21e48748b3be934961123b6962" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72027f21e48748b3be934961123b6962" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34245 } } }
I20260808 04:09:01.823268 2999 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72027f21e48748b3be934961123b6962 [sys.catalog]: This master's current role is: LEADER
I20260808 04:09:01.824425 3003 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:09:01.830080 3003 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:09:01.850569 3003 catalog_manager.cc:1347] Generated new cluster ID: ada6e75f95ca4389bbe0b0ca7dbce586
I20260808 04:09:01.850675 3003 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:09:01.851073 2992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:01.827666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54021 (local address 127.0.0.1:34245)
0808 04:09:01.827846 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:01.827852 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:01.833515 (+ 5663us) server_negotiation.cc:407] Connection header received
0808 04:09:01.833778 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:01.833783 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:01.833859 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:01.833932 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:01.836101 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:01.837164 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:01.839176 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:01.839898 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:01.840003 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:01.843551 (+ 3548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:09:01.843559 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:09:01.843565 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:09:01.843606 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:09:01.847429 (+ 3823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:09:01.848451 (+ 1022us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:09:01.848456 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:09:01.848459 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:09:01.848598 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:09:01.849173 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:09:01.849177 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:09:01.849184 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:09:01.849626 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:09:01.849879 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:01.850556 (+ 677us) server_negotiation.cc:299] Negotiation successful
0808 04:09:01.850735 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 04:09:01.859697 3003 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:09:01.860862 3003 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:09:01.869625 3003 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 72027f21e48748b3be934961123b6962: Generated new TSK 0
I20260808 04:09:01.870307 3003 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:09:01.876292 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 2873
I20260808 04:09:01.877467 2976 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:09:01.964236 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 2873
W20260808 04:09:01.982201 2852 connection.cc:535] client connection to 127.0.0.1:34245 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34245): Connection reset by peer (error 104)
2026-08-08T04:09:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2095 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786161826605655-5452-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:09:02 dist-test-slave-1l3l krb5kdc[3034](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:09:02 dist-test-slave-1l3l krb5kdc[3034](info): set up 2 sockets
Aug 08 04:09:02 dist-test-slave-1l3l krb5kdc[3034](info): commencing operation
krb5kdc: starting...
W20260808 04:09:03.154083 5452 mini_kdc.cc:121] Time spent starting KDC: real 1.153s user 0.001s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786161826605655-5452-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786161826605655-5452-0/krb5kdc/test-user.keytab.
Aug 08 04:09:03 dist-test-slave-1l3l krb5kdc[3034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:09:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:09:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786161826605655-5452-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786161826605655-5452-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 04:09:03.413291 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786161826605655-5452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786161826605655-5452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786161826605655-5452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40013
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786161826605655-5452-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40013
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:09:03.730738 3054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:09:03.731016 3054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:09:03.731158 3054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:09:03.738171 3054 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:09:03.738287 3054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:09:03.738365 3054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:09:03.738454 3054 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:09:03.738515 3054 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:09:03.747987 3054 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786161826605655-5452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786161826605655-5452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40013
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786161826605655-5452-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786161826605655-5452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:09:03.749111 3054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:09:03.750350 3054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:09:03.761179 3061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:09:03.761134 3062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:09:03.761214 3064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:09:03.762236 3054 server_base.cc:1034] running on GCE node
I20260808 04:09:03.762897 3054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:09:03.764144 3054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:09:03.765336 3054 hybrid_clock.cc:648] HybridClock initialized: now 1786162143765292 us; error 32 us; skew 500 ppm
I20260808 04:09:03.765637 3054 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:09:03 dist-test-slave-1l3l krb5kdc[3034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:09:03.768549 3054 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 04:09:03.769893 3054 webserver.cc:460] Webserver started at http://127.0.0.1:34687/ using document root <none> and password file <none>
I20260808 04:09:03.770511 3054 fs_manager.cc:359] Metadata directory not provided
I20260808 04:09:03.770619 3054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:09:03.770906 3054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:09:03.773713 3054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786161826605655-5452-0/minicluster-data/master-0/data/instance:
uuid: "a71acb420ad24356b8721b42adb4087a"
format_stamp: "Formatted at 2026-08-08 04:09:03 on dist-test-slave-1l3l"
server_key: "8a42f11a011a3b0f220e4b8e2d4e4082"
server_key_iv: "5c8ea28f5b0def58b2be81ff9555a723"
server_key_version: "encryptionkey@0"
I20260808 04:09:03.774416 3054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786161826605655-5452-0/minicluster-data/master-0/wal/instance:
uuid: "a71acb420ad24356b8721b42adb4087a"
format_stamp: "Formatted at 2026-08-08 04:09:03 on dist-test-slave-1l3l"
server_key: "8a42f11a011a3b0f220e4b8e2d4e4082"
server_key_iv: "5c8ea28f5b0def58b2be81ff9555a723"
server_key_version: "encryptionkey@0"
I20260808 04:09:03.779173 3054 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260808 04:09:03.782374 3071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:09:03.783599 3054 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:09:03.783762 3054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786161826605655-5452-0/minicluster-data/master-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786161826605655-5452-0/minicluster-data/master-0/wal
uuid: "a71acb420ad24356b8721b42adb4087a"
format_stamp: "Formatted at 2026-08-08 04:09:03 on dist-test-slave-1l3l"
server_key: "8a42f11a011a3b0f220e4b8e2d4e4082"
server_key_iv: "5c8ea28f5b0def58b2be81ff9555a723"
server_key_version: "encryptionkey@0"
I20260808 04:09:03.783922 3054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786161826605655-5452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786161826605655-5452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786161826605655-5452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:09:03.817801 3054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:09:03.821810 3054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:09:03.822058 3054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:09:03.838728 3054 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40013
I20260808 04:09:03.838768 3151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40013 every 8 connection(s)
I20260808 04:09:03.839991 3054 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
I20260808 04:09:03.843488 3152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:09:03.845831 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 3054
I20260808 04:09:03.846011 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786161826605655-5452-0/minicluster-data/master-0/wal/instance
I20260808 04:09:03.846536 5452 external_mini_cluster.cc:1428] Setting key a068db302b301125082461a407646aa8
I20260808 04:09:03.855233 3152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a71acb420ad24356b8721b42adb4087a: Bootstrap starting.
I20260808 04:09:03.858596 3152 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a71acb420ad24356b8721b42adb4087a: Neither blocks nor log segments found. Creating new log.
I20260808 04:09:03.859352 3155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:03.848346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40785 (local address 127.0.0.1:40013)
0808 04:09:03.849111 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:03.849122 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:03.849165 (+ 43us) server_negotiation.cc:407] Connection header received
0808 04:09:03.850764 (+ 1599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:03.850785 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:03.851137 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:03.851426 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:03.852010 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:03.853339 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:03.855158 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:03.855860 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:03.855991 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:03.857221 (+ 1230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:09:03.857228 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:09:03.857240 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:09:03.857730 (+ 490us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:09:03.857804 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:09:03.857828 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:09:03.857951 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:03.858208 (+ 257us) server_negotiation.cc:299] Negotiation successful
0808 04:09:03.858362 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":334,"threads_started":1}
I20260808 04:09:03.859678 3152 log.cc:824] T 00000000000000000000000000000000 P a71acb420ad24356b8721b42adb4087a: Log is configured to *not* fsync() on all Append() calls
I20260808 04:09:03.862437 3152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a71acb420ad24356b8721b42adb4087a: No bootstrap required, opened a new log
I20260808 04:09:03.862937 5452 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:09:03.870066 3152 raft_consensus.cc:348] T 00000000000000000000000000000000 P a71acb420ad24356b8721b42adb4087a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a71acb420ad24356b8721b42adb4087a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40013 } }
I20260808 04:09:03.870386 3152 raft_consensus.cc:374] T 00000000000000000000000000000000 P a71acb420ad24356b8721b42adb4087a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:09:03.870476 3152 raft_consensus.cc:729] T 00000000000000000000000000000000 P a71acb420ad24356b8721b42adb4087a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a71acb420ad24356b8721b42adb4087a, State: Initialized, Role: FOLLOWER
I20260808 04:09:03.871081 3152 consensus_queue.cc:260] T 00000000000000000000000000000000 P a71acb420ad24356b8721b42adb4087a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a71acb420ad24356b8721b42adb4087a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40013 } }
I20260808 04:09:03.871372 3152 raft_consensus.cc:388] T 00000000000000000000000000000000 P a71acb420ad24356b8721b42adb4087a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:09:03.871531 3152 raft_consensus.cc:482] T 00000000000000000000000000000000 P a71acb420ad24356b8721b42adb4087a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:09:03.871692 3152 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a71acb420ad24356b8721b42adb4087a [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:09:03.873219 3155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:03.866978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40229 (local address 127.0.0.1:40013)
0808 04:09:03.867139 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:03.867144 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:03.867493 (+ 349us) server_negotiation.cc:407] Connection header received
0808 04:09:03.867616 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:03.867621 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:03.867689 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:03.867817 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:03.868209 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:03.869319 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:03.870658 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:03.871289 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:03.871425 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:03.872532 (+ 1107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:09:03.872541 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:09:03.872546 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:09:03.872565 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:09:03.872608 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:09:03.872630 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:09:03.872753 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:03.872865 (+ 112us) server_negotiation.cc:299] Negotiation successful
0808 04:09:03.872911 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 04:09:03.874192 3152 raft_consensus.cc:504] T 00000000000000000000000000000000 P a71acb420ad24356b8721b42adb4087a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a71acb420ad24356b8721b42adb4087a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40013 } }
I20260808 04:09:03.874624 3152 leader_election.cc:302] T 00000000000000000000000000000000 P a71acb420ad24356b8721b42adb4087a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a71acb420ad24356b8721b42adb4087a; no voters:
I20260808 04:09:03.875123 3152 leader_election.cc:288] T 00000000000000000000000000000000 P a71acb420ad24356b8721b42adb4087a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:09:03.875216 3166 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a71acb420ad24356b8721b42adb4087a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:09:03.875905 3166 raft_consensus.cc:686] T 00000000000000000000000000000000 P a71acb420ad24356b8721b42adb4087a [term 1 LEADER]: Becoming Leader. State: Replica: a71acb420ad24356b8721b42adb4087a, State: Running, Role: LEADER
I20260808 04:09:03.876387 3166 consensus_queue.cc:237] T 00000000000000000000000000000000 P a71acb420ad24356b8721b42adb4087a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a71acb420ad24356b8721b42adb4087a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40013 } }
I20260808 04:09:03.876940 3152 sys_catalog.cc:564] T 00000000000000000000000000000000 P a71acb420ad24356b8721b42adb4087a [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:09:03.878813 3168 sys_catalog.cc:455] T 00000000000000000000000000000000 P a71acb420ad24356b8721b42adb4087a [sys.catalog]: SysCatalogTable state changed. Reason: New leader a71acb420ad24356b8721b42adb4087a. Latest consensus state: current_term: 1 leader_uuid: "a71acb420ad24356b8721b42adb4087a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a71acb420ad24356b8721b42adb4087a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40013 } } }
I20260808 04:09:03.879199 3168 sys_catalog.cc:458] T 00000000000000000000000000000000 P a71acb420ad24356b8721b42adb4087a [sys.catalog]: This master's current role is: LEADER
I20260808 04:09:03.880267 3167 sys_catalog.cc:455] T 00000000000000000000000000000000 P a71acb420ad24356b8721b42adb4087a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a71acb420ad24356b8721b42adb4087a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a71acb420ad24356b8721b42adb4087a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40013 } } }
I20260808 04:09:03.880589 3167 sys_catalog.cc:458] T 00000000000000000000000000000000 P a71acb420ad24356b8721b42adb4087a [sys.catalog]: This master's current role is: LEADER
I20260808 04:09:03.884649 3176 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:09:03.889940 3176 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:09:03.898119 3176 catalog_manager.cc:1347] Generated new cluster ID: 2949d6d96da5422cb22e1cf693c6ffe0
I20260808 04:09:03.898224 3176 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:09:03.906108 3176 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:09:03.907222 3176 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:09:03.913965 3176 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a71acb420ad24356b8721b42adb4087a: Generated new TSK 0
I20260808 04:09:03.914644 3176 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:09:03.922933 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 3054
W20260808 04:09:03.924034 3145 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:40785
W20260808 04:09:03.924692 5452 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 3054: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 04:09:03.924847 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 3054
W20260808 04:09:03.943251 3031 connection.cc:535] client connection to 127.0.0.1:40013 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40013): Connection reset by peer (error 104)
2026-08-08T04:09:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1960 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786161826605655-5452-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:09:04 dist-test-slave-1l3l krb5kdc[3194](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:09:04 dist-test-slave-1l3l krb5kdc[3194](info): set up 2 sockets
Aug 08 04:09:04 dist-test-slave-1l3l krb5kdc[3194](info): commencing operation
krb5kdc: starting...
W20260808 04:09:05.118772 5452 mini_kdc.cc:121] Time spent starting KDC: real 1.157s user 0.000s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786161826605655-5452-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786161826605655-5452-0/krb5kdc/test-user.keytab.
Aug 08 04:09:05 dist-test-slave-1l3l krb5kdc[3194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:09:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:09:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786161826605655-5452-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786161826605655-5452-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 04:09:05.382705 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786161826605655-5452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786161826605655-5452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786161826605655-5452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36853
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786161826605655-5452-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36853
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:09:05.698889 3210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:09:05.699162 3210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:09:05.699294 3210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:09:05.706208 3210 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:09:05.706318 3210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:09:05.706393 3210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:09:05.706486 3210 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:09:05.706542 3210 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:09:05.715806 3210 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786161826605655-5452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786161826605655-5452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36853
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786161826605655-5452-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786161826605655-5452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:09:05.716923 3210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:09:05.718139 3210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:09:05.729297 3215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:09:05.729290 3216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:09:05.729377 3218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:09:05.730023 3210 server_base.cc:1034] running on GCE node
I20260808 04:09:05.730696 3210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:09:05.731967 3210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:09:05.733183 3210 hybrid_clock.cc:648] HybridClock initialized: now 1786162145733130 us; error 41 us; skew 500 ppm
I20260808 04:09:05.733512 3210 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:09:05 dist-test-slave-1l3l krb5kdc[3194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162145, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:09:05.741385 3210 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 04:09:05.742705 3210 webserver.cc:460] Webserver started at http://127.0.0.1:44335/ using document root <none> and password file <none>
I20260808 04:09:05.743330 3210 fs_manager.cc:359] Metadata directory not provided
I20260808 04:09:05.743431 3210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:09:05.743686 3210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:09:05.746574 3210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786161826605655-5452-0/minicluster-data/master-0/data/instance:
uuid: "6a5455dd76534137910879344804aa25"
format_stamp: "Formatted at 2026-08-08 04:09:05 on dist-test-slave-1l3l"
server_key: "4662635644170a93b27ade23697cdf99"
server_key_iv: "a7460b92b73423fe085dd7a2a52c58f6"
server_key_version: "encryptionkey@0"
I20260808 04:09:05.747270 3210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786161826605655-5452-0/minicluster-data/master-0/wal/instance:
uuid: "6a5455dd76534137910879344804aa25"
format_stamp: "Formatted at 2026-08-08 04:09:05 on dist-test-slave-1l3l"
server_key: "4662635644170a93b27ade23697cdf99"
server_key_iv: "a7460b92b73423fe085dd7a2a52c58f6"
server_key_version: "encryptionkey@0"
I20260808 04:09:05.752220 3210 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260808 04:09:05.755379 3228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:09:05.756580 3210 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:09:05.756731 3210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786161826605655-5452-0/minicluster-data/master-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786161826605655-5452-0/minicluster-data/master-0/wal
uuid: "6a5455dd76534137910879344804aa25"
format_stamp: "Formatted at 2026-08-08 04:09:05 on dist-test-slave-1l3l"
server_key: "4662635644170a93b27ade23697cdf99"
server_key_iv: "a7460b92b73423fe085dd7a2a52c58f6"
server_key_version: "encryptionkey@0"
I20260808 04:09:05.756908 3210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786161826605655-5452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786161826605655-5452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786161826605655-5452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:09:05.762050 3210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:09:05.765552 3210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:09:05.765800 3210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:09:05.781987 3210 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36853
I20260808 04:09:05.782042 3312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36853 every 8 connection(s)
I20260808 04:09:05.783237 3210 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
I20260808 04:09:05.784859 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 3210
I20260808 04:09:05.785032 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786161826605655-5452-0/minicluster-data/master-0/wal/instance
I20260808 04:09:05.785460 5452 external_mini_cluster.cc:1428] Setting key 6c48497c6e3d20b99850f4094356f5b3
I20260808 04:09:05.787258 3313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:09:05.792557 3316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:05.787029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40927 (local address 127.0.0.1:36853)
0808 04:09:05.787819 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:05.787830 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:05.787870 (+ 40us) server_negotiation.cc:407] Connection header received
0808 04:09:05.789386 (+ 1516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:05.789407 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:05.789706 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:05.790025 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:05.791104 (+ 1079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:09:05.791111 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:09:05.791122 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:09:05.791146 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:09:05.791214 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:09:05.791238 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:09:05.791416 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:05.791519 (+ 103us) server_negotiation.cc:299] Negotiation successful
0808 04:09:05.791667 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":361,"threads_started":1}
I20260808 04:09:05.795922 5452 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:09:05.798643 3313 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a5455dd76534137910879344804aa25: Bootstrap starting.
I20260808 04:09:05.801690 3313 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a5455dd76534137910879344804aa25: Neither blocks nor log segments found. Creating new log.
I20260808 04:09:05.802119 3316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:05.799933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34979 (local address 127.0.0.1:36853)
0808 04:09:05.800079 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:05.800085 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:05.800475 (+ 390us) server_negotiation.cc:407] Connection header received
0808 04:09:05.800621 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:05.800626 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:05.800669 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:05.800743 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:05.801534 (+ 791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:09:05.801540 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:09:05.801545 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:09:05.801562 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:09:05.801596 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:09:05.801614 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:09:05.801707 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:05.801803 (+ 96us) server_negotiation.cc:299] Negotiation successful
0808 04:09:05.801854 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 04:09:05.802906 3313 log.cc:824] T 00000000000000000000000000000000 P 6a5455dd76534137910879344804aa25: Log is configured to *not* fsync() on all Append() calls
I20260808 04:09:05.805390 3313 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a5455dd76534137910879344804aa25: No bootstrap required, opened a new log
I20260808 04:09:05.812633 3313 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a5455dd76534137910879344804aa25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a5455dd76534137910879344804aa25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36853 } }
I20260808 04:09:05.812981 3313 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a5455dd76534137910879344804aa25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:09:05.813052 3313 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a5455dd76534137910879344804aa25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a5455dd76534137910879344804aa25, State: Initialized, Role: FOLLOWER
I20260808 04:09:05.813575 3313 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a5455dd76534137910879344804aa25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a5455dd76534137910879344804aa25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36853 } }
I20260808 04:09:05.813767 3313 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a5455dd76534137910879344804aa25 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:09:05.813862 3313 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a5455dd76534137910879344804aa25 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:09:05.813979 3313 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a5455dd76534137910879344804aa25 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:09:05.816008 3313 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a5455dd76534137910879344804aa25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a5455dd76534137910879344804aa25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36853 } }
I20260808 04:09:05.816419 3313 leader_election.cc:302] T 00000000000000000000000000000000 P 6a5455dd76534137910879344804aa25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a5455dd76534137910879344804aa25; no voters:
I20260808 04:09:05.816926 3313 leader_election.cc:288] T 00000000000000000000000000000000 P 6a5455dd76534137910879344804aa25 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:09:05.817045 3323 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a5455dd76534137910879344804aa25 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:09:05.817662 3323 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a5455dd76534137910879344804aa25 [term 1 LEADER]: Becoming Leader. State: Replica: 6a5455dd76534137910879344804aa25, State: Running, Role: LEADER
I20260808 04:09:05.818135 3323 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a5455dd76534137910879344804aa25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a5455dd76534137910879344804aa25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36853 } }
I20260808 04:09:05.818754 3313 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a5455dd76534137910879344804aa25 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:09:05.820636 3324 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a5455dd76534137910879344804aa25 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a5455dd76534137910879344804aa25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a5455dd76534137910879344804aa25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36853 } } }
I20260808 04:09:05.821033 3324 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a5455dd76534137910879344804aa25 [sys.catalog]: This master's current role is: LEADER
I20260808 04:09:05.822201 3326 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a5455dd76534137910879344804aa25 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a5455dd76534137910879344804aa25. Latest consensus state: current_term: 1 leader_uuid: "6a5455dd76534137910879344804aa25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a5455dd76534137910879344804aa25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36853 } } }
I20260808 04:09:05.822506 3326 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a5455dd76534137910879344804aa25 [sys.catalog]: This master's current role is: LEADER
I20260808 04:09:05.826761 3335 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:09:05.830623 3335 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:09:05.838600 3335 catalog_manager.cc:1347] Generated new cluster ID: b8408440dc8545da8bf6301e954753a4
I20260808 04:09:05.838696 3335 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:09:05.863217 3335 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:09:05.863605 3335 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:09:05.874132 3335 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a5455dd76534137910879344804aa25: Generated new TSK 0
I20260808 04:09:05.874837 3335 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:09:05.880366 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 3210
W20260808 04:09:05.881316 3306 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:40927
W20260808 04:09:05.881914 5452 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 3210: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 04:09:05.882033 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 3210
2026-08-08T04:09:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1961 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786161826605655-5452-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:09:05 dist-test-slave-1l3l krb5kdc[3362](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:09:05 dist-test-slave-1l3l krb5kdc[3362](info): set up 2 sockets
Aug 08 04:09:05 dist-test-slave-1l3l krb5kdc[3362](info): commencing operation
krb5kdc: starting...
W20260808 04:09:07.073168 5452 mini_kdc.cc:121] Time spent starting KDC: real 1.150s user 0.002s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786161826605655-5452-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786161826605655-5452-0/krb5kdc/test-user.keytab.
Aug 08 04:09:07 dist-test-slave-1l3l krb5kdc[3362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162147, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:09:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:09:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786161826605655-5452-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786161826605655-5452-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 04:09:07.346203 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786161826605655-5452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786161826605655-5452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786161826605655-5452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45105
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786161826605655-5452-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45105
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:09:07.664075 3378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:09:07.664361 3378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:09:07.664503 3378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:09:07.671355 3378 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:09:07.671463 3378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:09:07.671546 3378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:09:07.671629 3378 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:09:07.671691 3378 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:09:07.681152 3378 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786161826605655-5452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786161826605655-5452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45105
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786161826605655-5452-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786161826605655-5452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:09:07.682299 3378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:09:07.683542 3378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:09:07.694271 3387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:09:07.694293 3391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:09:07.694350 3385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:09:07.694864 3378 server_base.cc:1034] running on GCE node
I20260808 04:09:07.695627 3378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:09:07.696877 3378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:09:07.698078 3378 hybrid_clock.cc:648] HybridClock initialized: now 1786162147698031 us; error 43 us; skew 500 ppm
I20260808 04:09:07.698387 3378 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:09:07 dist-test-slave-1l3l krb5kdc[3362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162147, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:09:07.701440 3378 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 04:09:07.702714 3378 webserver.cc:460] Webserver started at http://127.0.0.1:34023/ using document root <none> and password file <none>
I20260808 04:09:07.703311 3378 fs_manager.cc:359] Metadata directory not provided
I20260808 04:09:07.703408 3378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:09:07.703645 3378 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:09:07.706353 3378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786161826605655-5452-0/minicluster-data/master-0/data/instance:
uuid: "71562286035d4e41bb98f6f258781f49"
format_stamp: "Formatted at 2026-08-08 04:09:07 on dist-test-slave-1l3l"
server_key: "bf611935a3e8c250cdac2be1832cf67d"
server_key_iv: "52fa4637438ab4b595dd45f81b5a85fd"
server_key_version: "encryptionkey@0"
I20260808 04:09:07.707039 3378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786161826605655-5452-0/minicluster-data/master-0/wal/instance:
uuid: "71562286035d4e41bb98f6f258781f49"
format_stamp: "Formatted at 2026-08-08 04:09:07 on dist-test-slave-1l3l"
server_key: "bf611935a3e8c250cdac2be1832cf67d"
server_key_iv: "52fa4637438ab4b595dd45f81b5a85fd"
server_key_version: "encryptionkey@0"
I20260808 04:09:07.711702 3378 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260808 04:09:07.714892 3401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:09:07.716127 3378 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:09:07.716276 3378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786161826605655-5452-0/minicluster-data/master-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786161826605655-5452-0/minicluster-data/master-0/wal
uuid: "71562286035d4e41bb98f6f258781f49"
format_stamp: "Formatted at 2026-08-08 04:09:07 on dist-test-slave-1l3l"
server_key: "bf611935a3e8c250cdac2be1832cf67d"
server_key_iv: "52fa4637438ab4b595dd45f81b5a85fd"
server_key_version: "encryptionkey@0"
I20260808 04:09:07.716442 3378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786161826605655-5452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786161826605655-5452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786161826605655-5452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:09:07.745119 3378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:09:07.748687 3378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:09:07.748948 3378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:09:07.765278 3378 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45105
I20260808 04:09:07.765300 3478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45105 every 8 connection(s)
I20260808 04:09:07.766549 3378 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
I20260808 04:09:07.770006 3484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:09:07.770339 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 3378
I20260808 04:09:07.770493 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786161826605655-5452-0/minicluster-data/master-0/wal/instance
I20260808 04:09:07.770928 5452 external_mini_cluster.cc:1428] Setting key 954b331f89c2e87ae78601cba906dc57
I20260808 04:09:07.783317 3484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71562286035d4e41bb98f6f258781f49: Bootstrap starting.
Aug 08 04:09:07 dist-test-slave-1l3l krb5kdc[3362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162147, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260808 04:09:07.786078 3484 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71562286035d4e41bb98f6f258781f49: Neither blocks nor log segments found. Creating new log.
I20260808 04:09:07.787231 3484 log.cc:824] T 00000000000000000000000000000000 P 71562286035d4e41bb98f6f258781f49: Log is configured to *not* fsync() on all Append() calls
I20260808 04:09:07.789958 3484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71562286035d4e41bb98f6f258781f49: No bootstrap required, opened a new log
I20260808 04:09:07.793655 3487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:07.772568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41767 (local address 127.0.0.1:45105)
0808 04:09:07.773355 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:07.773369 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:07.773418 (+ 49us) server_negotiation.cc:407] Connection header received
0808 04:09:07.776132 (+ 2714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:07.776159 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:07.776620 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:07.777039 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:07.778331 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:07.780291 (+ 1960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:07.781587 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:07.782582 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:07.782750 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:07.786597 (+ 3847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:09:07.786607 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:09:07.786623 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:09:07.786662 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:09:07.789796 (+ 3134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:09:07.790601 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:09:07.790607 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:09:07.790612 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:09:07.790776 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:09:07.791170 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:09:07.791173 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:09:07.791175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:09:07.791660 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:09:07.791921 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:07.792326 (+ 405us) server_negotiation.cc:299] Negotiation successful
0808 04:09:07.792678 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":325,"threads_started":1}
I20260808 04:09:07.797266 5452 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:09:07.798868 3484 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71562286035d4e41bb98f6f258781f49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71562286035d4e41bb98f6f258781f49" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45105 } }
I20260808 04:09:07.799269 3484 raft_consensus.cc:374] T 00000000000000000000000000000000 P 71562286035d4e41bb98f6f258781f49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:09:07.799360 3484 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71562286035d4e41bb98f6f258781f49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71562286035d4e41bb98f6f258781f49, State: Initialized, Role: FOLLOWER
I20260808 04:09:07.800010 3484 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71562286035d4e41bb98f6f258781f49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71562286035d4e41bb98f6f258781f49" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45105 } }
I20260808 04:09:07.800295 3484 raft_consensus.cc:388] T 00000000000000000000000000000000 P 71562286035d4e41bb98f6f258781f49 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:09:07.800441 3484 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71562286035d4e41bb98f6f258781f49 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:09:07.800604 3484 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71562286035d4e41bb98f6f258781f49 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:09:07.803259 3484 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71562286035d4e41bb98f6f258781f49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71562286035d4e41bb98f6f258781f49" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45105 } }
I20260808 04:09:07.803682 3484 leader_election.cc:302] T 00000000000000000000000000000000 P 71562286035d4e41bb98f6f258781f49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 71562286035d4e41bb98f6f258781f49; no voters:
I20260808 04:09:07.804270 3484 leader_election.cc:288] T 00000000000000000000000000000000 P 71562286035d4e41bb98f6f258781f49 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:09:07.804431 3499 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71562286035d4e41bb98f6f258781f49 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:09:07.806075 3499 raft_consensus.cc:686] T 00000000000000000000000000000000 P 71562286035d4e41bb98f6f258781f49 [term 1 LEADER]: Becoming Leader. State: Replica: 71562286035d4e41bb98f6f258781f49, State: Running, Role: LEADER
I20260808 04:09:07.806298 3484 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71562286035d4e41bb98f6f258781f49 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:09:07.806638 3499 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71562286035d4e41bb98f6f258781f49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71562286035d4e41bb98f6f258781f49" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45105 } }
I20260808 04:09:07.809648 3503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71562286035d4e41bb98f6f258781f49 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71562286035d4e41bb98f6f258781f49. Latest consensus state: current_term: 1 leader_uuid: "71562286035d4e41bb98f6f258781f49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71562286035d4e41bb98f6f258781f49" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45105 } } }
I20260808 04:09:07.810098 3503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71562286035d4e41bb98f6f258781f49 [sys.catalog]: This master's current role is: LEADER
I20260808 04:09:07.817517 3518 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:09:07.817588 3501 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71562286035d4e41bb98f6f258781f49 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "71562286035d4e41bb98f6f258781f49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71562286035d4e41bb98f6f258781f49" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45105 } } }
I20260808 04:09:07.818742 3487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:07.802103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36093 (local address 127.0.0.1:45105)
0808 04:09:07.802258 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:07.802263 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:07.802646 (+ 383us) server_negotiation.cc:407] Connection header received
0808 04:09:07.802751 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:07.802756 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:07.802821 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:07.802912 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:07.804533 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:07.805638 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:07.807062 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:07.807538 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:07.807673 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:07.810932 (+ 3259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:09:07.810939 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:09:07.810945 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:09:07.810982 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:09:07.814625 (+ 3643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:09:07.815488 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:09:07.815494 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:09:07.815498 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:09:07.815645 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:09:07.816128 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:09:07.816133 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:09:07.816137 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:09:07.816566 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:09:07.816824 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:07.818164 (+ 1340us) server_negotiation.cc:299] Negotiation successful
0808 04:09:07.818365 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 04:09:07.819890 3501 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71562286035d4e41bb98f6f258781f49 [sys.catalog]: This master's current role is: LEADER
I20260808 04:09:07.826046 3518 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:09:07.834580 3518 catalog_manager.cc:1347] Generated new cluster ID: a792bae85e904a2e8a067961c94d389b
I20260808 04:09:07.834697 3518 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:09:07.848237 3518 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:09:07.849524 3518 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:09:07.857900 3518 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 71562286035d4e41bb98f6f258781f49: Generated new TSK 0
I20260808 04:09:07.858698 3518 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:09:07.867738 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 3378
I20260808 04:09:07.868752 3473 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:09:07.975080 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 3378
2026-08-08T04:09:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2091 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786161826605655-5452-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:09:08 dist-test-slave-1l3l krb5kdc[3535](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:09:08 dist-test-slave-1l3l krb5kdc[3535](info): set up 2 sockets
Aug 08 04:09:08 dist-test-slave-1l3l krb5kdc[3535](info): commencing operation
krb5kdc: starting...
W20260808 04:09:09.172072 5452 mini_kdc.cc:121] Time spent starting KDC: real 1.157s user 0.002s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786161826605655-5452-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786161826605655-5452-0/krb5kdc/test-user.keytab.
Aug 08 04:09:09 dist-test-slave-1l3l krb5kdc[3535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162149, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:09:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:09:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786161826605655-5452-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786161826605655-5452-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 04:09:09.430840 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786161826605655-5452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786161826605655-5452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786161826605655-5452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41579
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786161826605655-5452-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41579
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:09:09.747133 3551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:09:09.747396 3551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:09:09.747517 3551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:09:09.754590 3551 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:09:09.754684 3551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:09:09.754756 3551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:09:09.754838 3551 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:09:09.754894 3551 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:09:09.764518 3551 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786161826605655-5452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786161826605655-5452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41579
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786161826605655-5452-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786161826605655-5452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:09:09.765676 3551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:09:09.766908 3551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:09:09.778044 3558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:09:09.778004 3562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:09:09.778077 3560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:09:09.779014 3551 server_base.cc:1034] running on GCE node
I20260808 04:09:09.779810 3551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:09:09.781095 3551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:09:09.782284 3551 hybrid_clock.cc:648] HybridClock initialized: now 1786162149782249 us; error 29 us; skew 500 ppm
I20260808 04:09:09.782608 3551 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:09:09 dist-test-slave-1l3l krb5kdc[3535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162149, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:09:09.785534 3551 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 04:09:09.786948 3551 webserver.cc:460] Webserver started at http://127.0.0.1:43379/ using document root <none> and password file <none>
I20260808 04:09:09.787585 3551 fs_manager.cc:359] Metadata directory not provided
I20260808 04:09:09.787693 3551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:09:09.788014 3551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:09:09.790807 3551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786161826605655-5452-0/minicluster-data/master-0/data/instance:
uuid: "f2f7e596535a48e08e4a73b597518b89"
format_stamp: "Formatted at 2026-08-08 04:09:09 on dist-test-slave-1l3l"
server_key: "ee091521eab07d171df2547a54f81c95"
server_key_iv: "bf8e3cfe52d9e03d49f0dbdf49f7d1d3"
server_key_version: "encryptionkey@0"
I20260808 04:09:09.791525 3551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786161826605655-5452-0/minicluster-data/master-0/wal/instance:
uuid: "f2f7e596535a48e08e4a73b597518b89"
format_stamp: "Formatted at 2026-08-08 04:09:09 on dist-test-slave-1l3l"
server_key: "ee091521eab07d171df2547a54f81c95"
server_key_iv: "bf8e3cfe52d9e03d49f0dbdf49f7d1d3"
server_key_version: "encryptionkey@0"
I20260808 04:09:09.796334 3551 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260808 04:09:09.799463 3572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:09:09.800662 3551 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 04:09:09.800809 3551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786161826605655-5452-0/minicluster-data/master-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786161826605655-5452-0/minicluster-data/master-0/wal
uuid: "f2f7e596535a48e08e4a73b597518b89"
format_stamp: "Formatted at 2026-08-08 04:09:09 on dist-test-slave-1l3l"
server_key: "ee091521eab07d171df2547a54f81c95"
server_key_iv: "bf8e3cfe52d9e03d49f0dbdf49f7d1d3"
server_key_version: "encryptionkey@0"
I20260808 04:09:09.800976 3551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786161826605655-5452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786161826605655-5452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786161826605655-5452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:09:09.814944 3551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:09:09.818439 3551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:09:09.818681 3551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:09:09.835821 3551 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41579
I20260808 04:09:09.835861 3654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41579 every 8 connection(s)
I20260808 04:09:09.837129 3551 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
I20260808 04:09:09.840612 3656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:09:09.843789 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 3551
I20260808 04:09:09.843967 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786161826605655-5452-0/minicluster-data/master-0/wal/instance
I20260808 04:09:09.844527 5452 external_mini_cluster.cc:1428] Setting key c4233f0bc09a573d37d87e507ed236bf
I20260808 04:09:09.852340 3656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2f7e596535a48e08e4a73b597518b89: Bootstrap starting.
I20260808 04:09:09.855684 3656 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2f7e596535a48e08e4a73b597518b89: Neither blocks nor log segments found. Creating new log.
I20260808 04:09:09.856889 3656 log.cc:824] T 00000000000000000000000000000000 P f2f7e596535a48e08e4a73b597518b89: Log is configured to *not* fsync() on all Append() calls
I20260808 04:09:09.857288 3659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:09.846284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58291 (local address 127.0.0.1:41579)
0808 04:09:09.846989 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:09.847000 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:09.847041 (+ 41us) server_negotiation.cc:407] Connection header received
0808 04:09:09.848693 (+ 1652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:09.848714 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:09.849088 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:09.849392 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:09.849985 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:09.851289 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:09.853054 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:09.853698 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:09.853841 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:09.855030 (+ 1189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:09:09.855037 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:09:09.855049 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:09:09.855559 (+ 510us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:09:09.855642 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:09:09.855676 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:09:09.855828 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:09.856023 (+ 195us) server_negotiation.cc:299] Negotiation successful
0808 04:09:09.856257 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":330,"threads_started":1}
I20260808 04:09:09.859963 3656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2f7e596535a48e08e4a73b597518b89: No bootstrap required, opened a new log
I20260808 04:09:09.861037 5452 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:09:09.869553 3656 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2f7e596535a48e08e4a73b597518b89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2f7e596535a48e08e4a73b597518b89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41579 } }
I20260808 04:09:09.870002 3656 raft_consensus.cc:374] T 00000000000000000000000000000000 P f2f7e596535a48e08e4a73b597518b89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:09:09.870128 3656 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2f7e596535a48e08e4a73b597518b89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2f7e596535a48e08e4a73b597518b89, State: Initialized, Role: FOLLOWER
I20260808 04:09:09.870780 3656 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2f7e596535a48e08e4a73b597518b89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2f7e596535a48e08e4a73b597518b89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41579 } }
I20260808 04:09:09.871011 3656 raft_consensus.cc:388] T 00000000000000000000000000000000 P f2f7e596535a48e08e4a73b597518b89 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:09:09.871143 3656 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2f7e596535a48e08e4a73b597518b89 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:09:09.871289 3656 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2f7e596535a48e08e4a73b597518b89 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:09:09.872366 3659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:09.865917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54035 (local address 127.0.0.1:41579)
0808 04:09:09.866085 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:09.866091 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:09.866536 (+ 445us) server_negotiation.cc:407] Connection header received
0808 04:09:09.866665 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:09.866670 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:09.866751 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:09.866863 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:09.867294 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:09.868453 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:09.869761 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:09.870401 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:09.870520 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:09.871607 (+ 1087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:09:09.871613 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:09:09.871617 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:09:09.871636 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:09:09.871678 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:09:09.871698 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:09:09.871822 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:09.871946 (+ 124us) server_negotiation.cc:299] Negotiation successful
0808 04:09:09.872042 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 04:09:09.874051 3656 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2f7e596535a48e08e4a73b597518b89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2f7e596535a48e08e4a73b597518b89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41579 } }
I20260808 04:09:09.874477 3656 leader_election.cc:302] T 00000000000000000000000000000000 P f2f7e596535a48e08e4a73b597518b89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f2f7e596535a48e08e4a73b597518b89; no voters:
I20260808 04:09:09.874989 3656 leader_election.cc:288] T 00000000000000000000000000000000 P f2f7e596535a48e08e4a73b597518b89 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:09:09.875082 3668 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f2f7e596535a48e08e4a73b597518b89 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:09:09.875730 3668 raft_consensus.cc:686] T 00000000000000000000000000000000 P f2f7e596535a48e08e4a73b597518b89 [term 1 LEADER]: Becoming Leader. State: Replica: f2f7e596535a48e08e4a73b597518b89, State: Running, Role: LEADER
I20260808 04:09:09.876267 3668 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2f7e596535a48e08e4a73b597518b89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2f7e596535a48e08e4a73b597518b89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41579 } }
I20260808 04:09:09.877125 3656 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2f7e596535a48e08e4a73b597518b89 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:09:09.878921 3671 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2f7e596535a48e08e4a73b597518b89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2f7e596535a48e08e4a73b597518b89. Latest consensus state: current_term: 1 leader_uuid: "f2f7e596535a48e08e4a73b597518b89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2f7e596535a48e08e4a73b597518b89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41579 } } }
I20260808 04:09:09.878921 3670 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2f7e596535a48e08e4a73b597518b89 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f2f7e596535a48e08e4a73b597518b89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2f7e596535a48e08e4a73b597518b89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41579 } } }
I20260808 04:09:09.879633 3670 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2f7e596535a48e08e4a73b597518b89 [sys.catalog]: This master's current role is: LEADER
I20260808 04:09:09.880199 3671 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2f7e596535a48e08e4a73b597518b89 [sys.catalog]: This master's current role is: LEADER
I20260808 04:09:09.880905 3677 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:09:09.885747 3677 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:09:09.894708 3677 catalog_manager.cc:1347] Generated new cluster ID: 6c4f6c58ae334f3d8996c3708d42f1bf
I20260808 04:09:09.894809 3677 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:09:09.916198 3677 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:09:09.917387 3677 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:09:09.924237 3677 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f2f7e596535a48e08e4a73b597518b89: Generated new TSK 0
I20260808 04:09:09.924921 3677 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:09:09.935451 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 3551
W20260808 04:09:09.936471 3650 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:58291
W20260808 04:09:09.937115 5452 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 3551: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 04:09:09.937247 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 3551
2026-08-08T04:09:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1964 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786161826605655-5452-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:09:10 dist-test-slave-1l3l krb5kdc[3701](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:09:10 dist-test-slave-1l3l krb5kdc[3701](info): set up 2 sockets
Aug 08 04:09:10 dist-test-slave-1l3l krb5kdc[3701](info): commencing operation
krb5kdc: starting...
W20260808 04:09:11.141202 5452 mini_kdc.cc:121] Time spent starting KDC: real 1.162s user 0.004s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786161826605655-5452-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786161826605655-5452-0/krb5kdc/test-user.keytab.
Aug 08 04:09:11 dist-test-slave-1l3l krb5kdc[3701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162151, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:09:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:09:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786161826605655-5452-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786161826605655-5452-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 04:09:11.425100 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786161826605655-5452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786161826605655-5452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786161826605655-5452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41983
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786161826605655-5452-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41983
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:09:11.742265 3717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:09:11.742520 3717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:09:11.742650 3717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:09:11.749523 3717 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:09:11.749616 3717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:09:11.749694 3717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:09:11.749770 3717 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:09:11.749828 3717 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:09:11.758966 3717 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786161826605655-5452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786161826605655-5452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41983
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786161826605655-5452-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786161826605655-5452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:09:11.760128 3717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:09:11.761298 3717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:09:11.772131 3726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:09:11.772325 3729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:09:11.772408 3724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:09:11.772919 3717 server_base.cc:1034] running on GCE node
I20260808 04:09:11.773554 3717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:09:11.774742 3717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:09:11.775940 3717 hybrid_clock.cc:648] HybridClock initialized: now 1786162151775905 us; error 28 us; skew 500 ppm
I20260808 04:09:11.776249 3717 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:09:11 dist-test-slave-1l3l krb5kdc[3701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162151, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:09:11.784559 3717 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 04:09:11.785938 3717 webserver.cc:460] Webserver started at http://127.0.0.1:46185/ using document root <none> and password file <none>
I20260808 04:09:11.786545 3717 fs_manager.cc:359] Metadata directory not provided
I20260808 04:09:11.786643 3717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:09:11.786895 3717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:09:11.789716 3717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786161826605655-5452-0/minicluster-data/master-0/data/instance:
uuid: "8b93a2e2d4d8420591c5d51c3eb4e4aa"
format_stamp: "Formatted at 2026-08-08 04:09:11 on dist-test-slave-1l3l"
server_key: "6daebad09df970c51e5f0ef1c924b2db"
server_key_iv: "a06aff4d2eaabb6df33e3bd8b490b56e"
server_key_version: "encryptionkey@0"
I20260808 04:09:11.790396 3717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786161826605655-5452-0/minicluster-data/master-0/wal/instance:
uuid: "8b93a2e2d4d8420591c5d51c3eb4e4aa"
format_stamp: "Formatted at 2026-08-08 04:09:11 on dist-test-slave-1l3l"
server_key: "6daebad09df970c51e5f0ef1c924b2db"
server_key_iv: "a06aff4d2eaabb6df33e3bd8b490b56e"
server_key_version: "encryptionkey@0"
I20260808 04:09:11.795085 3717 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260808 04:09:11.798205 3739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:09:11.799381 3717 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:09:11.799536 3717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786161826605655-5452-0/minicluster-data/master-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786161826605655-5452-0/minicluster-data/master-0/wal
uuid: "8b93a2e2d4d8420591c5d51c3eb4e4aa"
format_stamp: "Formatted at 2026-08-08 04:09:11 on dist-test-slave-1l3l"
server_key: "6daebad09df970c51e5f0ef1c924b2db"
server_key_iv: "a06aff4d2eaabb6df33e3bd8b490b56e"
server_key_version: "encryptionkey@0"
I20260808 04:09:11.799729 3717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786161826605655-5452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786161826605655-5452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786161826605655-5452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:09:11.805102 3717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:09:11.808616 3717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:09:11.808878 3717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:09:11.825719 3717 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41983
I20260808 04:09:11.825757 3806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41983 every 8 connection(s)
I20260808 04:09:11.827015 3717 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
I20260808 04:09:11.829147 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 3717
I20260808 04:09:11.829317 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786161826605655-5452-0/minicluster-data/master-0/wal/instance
I20260808 04:09:11.829757 5452 external_mini_cluster.cc:1428] Setting key 478490fab7d35aef347524dbe30e98f1
I20260808 04:09:11.830791 3808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:09:11.837942 3813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:11.831545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33843 (local address 127.0.0.1:41983)
0808 04:09:11.832427 (+ 882us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:11.832441 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:11.832492 (+ 51us) server_negotiation.cc:407] Connection header received
0808 04:09:11.834188 (+ 1696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:11.834215 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:11.834653 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:11.835080 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:11.836086 (+ 1006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:09:11.836095 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:09:11.836110 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:09:11.836127 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:09:11.836219 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:09:11.836244 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:09:11.836480 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:11.836605 (+ 125us) server_negotiation.cc:299] Negotiation successful
0808 04:09:11.836791 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":379,"threads_started":1}
I20260808 04:09:11.841288 5452 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:09:11.844933 3808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b93a2e2d4d8420591c5d51c3eb4e4aa: Bootstrap starting.
I20260808 04:09:11.848620 3808 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b93a2e2d4d8420591c5d51c3eb4e4aa: Neither blocks nor log segments found. Creating new log.
I20260808 04:09:11.849889 3808 log.cc:824] T 00000000000000000000000000000000 P 8b93a2e2d4d8420591c5d51c3eb4e4aa: Log is configured to *not* fsync() on all Append() calls
I20260808 04:09:11.849959 3813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:11.846390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33333 (local address 127.0.0.1:41983)
0808 04:09:11.846588 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:11.846595 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:11.846996 (+ 401us) server_negotiation.cc:407] Connection header received
0808 04:09:11.847149 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:11.847152 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:11.847195 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:11.847267 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:11.849331 (+ 2064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:09:11.849335 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:09:11.849338 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:09:11.849352 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:09:11.849390 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:09:11.849405 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:09:11.849489 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:11.849647 (+ 158us) server_negotiation.cc:299] Negotiation successful
0808 04:09:11.849694 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 04:09:11.852581 3808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b93a2e2d4d8420591c5d51c3eb4e4aa: No bootstrap required, opened a new log
I20260808 04:09:11.860121 3808 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b93a2e2d4d8420591c5d51c3eb4e4aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b93a2e2d4d8420591c5d51c3eb4e4aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41983 } }
I20260808 04:09:11.860453 3808 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b93a2e2d4d8420591c5d51c3eb4e4aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:09:11.860544 3808 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b93a2e2d4d8420591c5d51c3eb4e4aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b93a2e2d4d8420591c5d51c3eb4e4aa, State: Initialized, Role: FOLLOWER
I20260808 04:09:11.861080 3808 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b93a2e2d4d8420591c5d51c3eb4e4aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b93a2e2d4d8420591c5d51c3eb4e4aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41983 } }
I20260808 04:09:11.861330 3808 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8b93a2e2d4d8420591c5d51c3eb4e4aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:09:11.861459 3808 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b93a2e2d4d8420591c5d51c3eb4e4aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:09:11.861611 3808 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b93a2e2d4d8420591c5d51c3eb4e4aa [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:09:11.863623 3808 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b93a2e2d4d8420591c5d51c3eb4e4aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b93a2e2d4d8420591c5d51c3eb4e4aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41983 } }
I20260808 04:09:11.864084 3808 leader_election.cc:302] T 00000000000000000000000000000000 P 8b93a2e2d4d8420591c5d51c3eb4e4aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b93a2e2d4d8420591c5d51c3eb4e4aa; no voters:
I20260808 04:09:11.864619 3808 leader_election.cc:288] T 00000000000000000000000000000000 P 8b93a2e2d4d8420591c5d51c3eb4e4aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:09:11.864709 3824 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b93a2e2d4d8420591c5d51c3eb4e4aa [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:09:11.865738 3824 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b93a2e2d4d8420591c5d51c3eb4e4aa [term 1 LEADER]: Becoming Leader. State: Replica: 8b93a2e2d4d8420591c5d51c3eb4e4aa, State: Running, Role: LEADER
I20260808 04:09:11.866204 3824 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b93a2e2d4d8420591c5d51c3eb4e4aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b93a2e2d4d8420591c5d51c3eb4e4aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41983 } }
I20260808 04:09:11.866546 3808 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b93a2e2d4d8420591c5d51c3eb4e4aa [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:09:11.868628 3830 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b93a2e2d4d8420591c5d51c3eb4e4aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b93a2e2d4d8420591c5d51c3eb4e4aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b93a2e2d4d8420591c5d51c3eb4e4aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41983 } } }
I20260808 04:09:11.868850 3831 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b93a2e2d4d8420591c5d51c3eb4e4aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b93a2e2d4d8420591c5d51c3eb4e4aa. Latest consensus state: current_term: 1 leader_uuid: "8b93a2e2d4d8420591c5d51c3eb4e4aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b93a2e2d4d8420591c5d51c3eb4e4aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41983 } } }
I20260808 04:09:11.869109 3830 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b93a2e2d4d8420591c5d51c3eb4e4aa [sys.catalog]: This master's current role is: LEADER
I20260808 04:09:11.869170 3831 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b93a2e2d4d8420591c5d51c3eb4e4aa [sys.catalog]: This master's current role is: LEADER
I20260808 04:09:11.874868 3837 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:09:11.880306 3837 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:09:11.889394 3837 catalog_manager.cc:1347] Generated new cluster ID: 9f421c24368a4e5daf7fc63c8c864875
I20260808 04:09:11.889492 3837 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:09:11.922324 3837 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:09:11.922711 3837 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:09:11.935281 3837 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8b93a2e2d4d8420591c5d51c3eb4e4aa: Generated new TSK 0
I20260808 04:09:11.936234 3837 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:09:11.952286 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 3717
W20260808 04:09:11.953253 3800 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:33843
W20260808 04:09:11.953909 5452 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 3717: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 04:09:11.954039 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 3717
2026-08-08T04:09:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (2018 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786161826605655-5452-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:09:12 dist-test-slave-1l3l krb5kdc[3852](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:09:12 dist-test-slave-1l3l krb5kdc[3852](info): set up 2 sockets
Aug 08 04:09:12 dist-test-slave-1l3l krb5kdc[3852](info): commencing operation
krb5kdc: starting...
W20260808 04:09:13.158484 5452 mini_kdc.cc:121] Time spent starting KDC: real 1.161s user 0.000s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786161826605655-5452-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786161826605655-5452-0/krb5kdc/test-user.keytab.
Aug 08 04:09:13 dist-test-slave-1l3l krb5kdc[3852](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162153, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:09:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:09:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786161826605655-5452-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786161826605655-5452-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 04:09:13.436520 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786161826605655-5452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786161826605655-5452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786161826605655-5452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46823
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786161826605655-5452-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46823
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:09:13.756139 3870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:09:13.756386 3870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:09:13.756513 3870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:09:13.763741 3870 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:09:13.763854 3870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:09:13.763929 3870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:09:13.764017 3870 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:09:13.764079 3870 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:09:13.773350 3870 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786161826605655-5452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786161826605655-5452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46823
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786161826605655-5452-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786161826605655-5452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:09:13.774494 3870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:09:13.775823 3870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:09:13.786727 3879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:09:13.786808 3880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:09:13.786769 3882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:09:13.787490 3870 server_base.cc:1034] running on GCE node
I20260808 04:09:13.788188 3870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:09:13.789400 3870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:09:13.790589 3870 hybrid_clock.cc:648] HybridClock initialized: now 1786162153790554 us; error 33 us; skew 500 ppm
I20260808 04:09:13.790905 3870 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:09:13 dist-test-slave-1l3l krb5kdc[3852](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162153, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:09:13.793759 3870 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 04:09:13.795140 3870 webserver.cc:460] Webserver started at http://127.0.0.1:34857/ using document root <none> and password file <none>
I20260808 04:09:13.795816 3870 fs_manager.cc:359] Metadata directory not provided
I20260808 04:09:13.795943 3870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:09:13.796218 3870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:09:13.798995 3870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786161826605655-5452-0/minicluster-data/master-0/data/instance:
uuid: "939e2f65bcd44143b8218ae3410cdf01"
format_stamp: "Formatted at 2026-08-08 04:09:13 on dist-test-slave-1l3l"
server_key: "3b50b4eef94242b0e1a19fe0d2016794"
server_key_iv: "f97cfa892ae10a7fd77ea72e337db981"
server_key_version: "encryptionkey@0"
I20260808 04:09:13.799751 3870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786161826605655-5452-0/minicluster-data/master-0/wal/instance:
uuid: "939e2f65bcd44143b8218ae3410cdf01"
format_stamp: "Formatted at 2026-08-08 04:09:13 on dist-test-slave-1l3l"
server_key: "3b50b4eef94242b0e1a19fe0d2016794"
server_key_iv: "f97cfa892ae10a7fd77ea72e337db981"
server_key_version: "encryptionkey@0"
I20260808 04:09:13.804410 3870 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260808 04:09:13.807514 3892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:09:13.808723 3870 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 04:09:13.808881 3870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786161826605655-5452-0/minicluster-data/master-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786161826605655-5452-0/minicluster-data/master-0/wal
uuid: "939e2f65bcd44143b8218ae3410cdf01"
format_stamp: "Formatted at 2026-08-08 04:09:13 on dist-test-slave-1l3l"
server_key: "3b50b4eef94242b0e1a19fe0d2016794"
server_key_iv: "f97cfa892ae10a7fd77ea72e337db981"
server_key_version: "encryptionkey@0"
I20260808 04:09:13.809042 3870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786161826605655-5452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786161826605655-5452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786161826605655-5452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:09:13.836223 3870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:09:13.839891 3870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:09:13.840135 3870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:09:13.856828 3870 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46823
I20260808 04:09:13.856858 3963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46823 every 8 connection(s)
I20260808 04:09:13.858086 3870 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
I20260808 04:09:13.861675 3964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:09:13.861708 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 3870
I20260808 04:09:13.861892 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786161826605655-5452-0/minicluster-data/master-0/wal/instance
I20260808 04:09:13.862476 5452 external_mini_cluster.cc:1428] Setting key 117a9ec4d368689acb8bb5caf82b4dbe
I20260808 04:09:13.875223 3964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 939e2f65bcd44143b8218ae3410cdf01: Bootstrap starting.
Aug 08 04:09:13 dist-test-slave-1l3l krb5kdc[3852](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162153, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260808 04:09:13.878273 3964 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 939e2f65bcd44143b8218ae3410cdf01: Neither blocks nor log segments found. Creating new log.
I20260808 04:09:13.879390 3964 log.cc:824] T 00000000000000000000000000000000 P 939e2f65bcd44143b8218ae3410cdf01: Log is configured to *not* fsync() on all Append() calls
I20260808 04:09:13.882045 3964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 939e2f65bcd44143b8218ae3410cdf01: No bootstrap required, opened a new log
I20260808 04:09:13.884739 3968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:13.864287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.111:59441 (local address 127.0.0.1:46823)
0808 04:09:13.865041 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:13.865052 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:13.865092 (+ 40us) server_negotiation.cc:407] Connection header received
0808 04:09:13.866550 (+ 1458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:13.866570 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:13.866924 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:13.867221 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:13.868955 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:13.870155 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:13.871874 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:13.872576 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:13.872713 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:13.877570 (+ 4857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:09:13.877578 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:09:13.877589 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:09:13.877623 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:09:13.880910 (+ 3287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:09:13.881650 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:09:13.881656 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:09:13.881661 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:09:13.881811 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:09:13.882261 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:09:13.882265 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:09:13.882267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:09:13.882748 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:09:13.882993 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:13.883471 (+ 478us) server_negotiation.cc:299] Negotiation successful
0808 04:09:13.883776 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":338,"threads_started":1}
I20260808 04:09:13.888211 5452 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:09:13.890446 3964 raft_consensus.cc:348] T 00000000000000000000000000000000 P 939e2f65bcd44143b8218ae3410cdf01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "939e2f65bcd44143b8218ae3410cdf01" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46823 } }
I20260808 04:09:13.890863 3964 raft_consensus.cc:374] T 00000000000000000000000000000000 P 939e2f65bcd44143b8218ae3410cdf01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:09:13.890976 3964 raft_consensus.cc:729] T 00000000000000000000000000000000 P 939e2f65bcd44143b8218ae3410cdf01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 939e2f65bcd44143b8218ae3410cdf01, State: Initialized, Role: FOLLOWER
I20260808 04:09:13.891590 3964 consensus_queue.cc:260] T 00000000000000000000000000000000 P 939e2f65bcd44143b8218ae3410cdf01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "939e2f65bcd44143b8218ae3410cdf01" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46823 } }
I20260808 04:09:13.891917 3964 raft_consensus.cc:388] T 00000000000000000000000000000000 P 939e2f65bcd44143b8218ae3410cdf01 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:09:13.892068 3964 raft_consensus.cc:482] T 00000000000000000000000000000000 P 939e2f65bcd44143b8218ae3410cdf01 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:09:13.892237 3964 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 939e2f65bcd44143b8218ae3410cdf01 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:09:13.894555 3964 raft_consensus.cc:504] T 00000000000000000000000000000000 P 939e2f65bcd44143b8218ae3410cdf01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "939e2f65bcd44143b8218ae3410cdf01" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46823 } }
I20260808 04:09:13.895064 3964 leader_election.cc:302] T 00000000000000000000000000000000 P 939e2f65bcd44143b8218ae3410cdf01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 939e2f65bcd44143b8218ae3410cdf01; no voters:
I20260808 04:09:13.895731 3964 leader_election.cc:288] T 00000000000000000000000000000000 P 939e2f65bcd44143b8218ae3410cdf01 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:09:13.895829 3978 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 939e2f65bcd44143b8218ae3410cdf01 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:09:13.897714 3978 raft_consensus.cc:686] T 00000000000000000000000000000000 P 939e2f65bcd44143b8218ae3410cdf01 [term 1 LEADER]: Becoming Leader. State: Replica: 939e2f65bcd44143b8218ae3410cdf01, State: Running, Role: LEADER
I20260808 04:09:13.897831 3964 sys_catalog.cc:564] T 00000000000000000000000000000000 P 939e2f65bcd44143b8218ae3410cdf01 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:09:13.898314 3978 consensus_queue.cc:237] T 00000000000000000000000000000000 P 939e2f65bcd44143b8218ae3410cdf01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "939e2f65bcd44143b8218ae3410cdf01" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46823 } }
I20260808 04:09:13.900766 3982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 939e2f65bcd44143b8218ae3410cdf01 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 939e2f65bcd44143b8218ae3410cdf01. Latest consensus state: current_term: 1 leader_uuid: "939e2f65bcd44143b8218ae3410cdf01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "939e2f65bcd44143b8218ae3410cdf01" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46823 } } }
I20260808 04:09:13.901618 3982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 939e2f65bcd44143b8218ae3410cdf01 [sys.catalog]: This master's current role is: LEADER
I20260808 04:09:13.901320 3981 sys_catalog.cc:455] T 00000000000000000000000000000000 P 939e2f65bcd44143b8218ae3410cdf01 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "939e2f65bcd44143b8218ae3410cdf01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "939e2f65bcd44143b8218ae3410cdf01" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46823 } } }
I20260808 04:09:13.902105 3981 sys_catalog.cc:458] T 00000000000000000000000000000000 P 939e2f65bcd44143b8218ae3410cdf01 [sys.catalog]: This master's current role is: LEADER
I20260808 04:09:13.909093 3988 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:09:13.911975 3968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:13.893331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.111:52429 (local address 127.0.0.1:46823)
0808 04:09:13.893508 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:13.893514 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:13.895217 (+ 1703us) server_negotiation.cc:407] Connection header received
0808 04:09:13.895427 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:13.895431 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:13.895502 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:13.895605 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:13.897427 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:13.898602 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:13.900537 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:13.901294 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:13.901427 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:13.904718 (+ 3291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:09:13.904725 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:09:13.904730 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:09:13.904770 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:09:13.908532 (+ 3762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:09:13.909450 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:09:13.909456 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:09:13.909460 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:09:13.909612 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:09:13.910225 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:09:13.910230 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:09:13.910234 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:09:13.910651 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:09:13.910954 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:13.911365 (+ 411us) server_negotiation.cc:299] Negotiation successful
0808 04:09:13.911553 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 04:09:13.915328 3988 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:09:13.923404 3988 catalog_manager.cc:1347] Generated new cluster ID: 21cec0a566c04c21ae94e99e5d70d6c2
I20260808 04:09:13.923499 3988 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:09:13.956081 3988 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:09:13.957809 3988 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:09:13.965171 3988 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 939e2f65bcd44143b8218ae3410cdf01: Generated new TSK 0
I20260808 04:09:13.966064 3988 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:09:13.973445 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 3870
I20260808 04:09:13.975021 3958 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:09:14.065228 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 3870
W20260808 04:09:14.087098 3850 connection.cc:535] client connection to 127.0.0.1:46823 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46823): Connection reset by peer (error 104)
2026-08-08T04:09:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2108 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786161826605655-5452-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:09:14 dist-test-slave-1l3l krb5kdc[4019](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:09:14 dist-test-slave-1l3l krb5kdc[4019](info): set up 2 sockets
Aug 08 04:09:14 dist-test-slave-1l3l krb5kdc[4019](info): commencing operation
krb5kdc: starting...
W20260808 04:09:15.265388 5452 mini_kdc.cc:121] Time spent starting KDC: real 1.160s user 0.005s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786161826605655-5452-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786161826605655-5452-0/krb5kdc/test-user.keytab.
Aug 08 04:09:15 dist-test-slave-1l3l krb5kdc[4019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162155, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:09:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:09:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786161826605655-5452-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786161826605655-5452-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 04:09:15.532665 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786161826605655-5452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786161826605655-5452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786161826605655-5452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36067
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786161826605655-5452-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36067
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:09:15.853416 4042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:09:15.853673 4042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:09:15.853790 4042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:09:15.860715 4042 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:09:15.860814 4042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:09:15.860869 4042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:09:15.860953 4042 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:09:15.861014 4042 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:09:15.870348 4042 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786161826605655-5452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786161826605655-5452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36067
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786161826605655-5452-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786161826605655-5452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:09:15.871501 4042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:09:15.872722 4042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:09:15.883682 4047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:09:15.883613 4048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:09:15.883643 4051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:09:15.884441 4042 server_base.cc:1034] running on GCE node
I20260808 04:09:15.885133 4042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:09:15.886368 4042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:09:15.887559 4042 hybrid_clock.cc:648] HybridClock initialized: now 1786162155887523 us; error 28 us; skew 500 ppm
I20260808 04:09:15.887909 4042 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:09:15 dist-test-slave-1l3l krb5kdc[4019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162155, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:09:15.890743 4042 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 04:09:15.892093 4042 webserver.cc:460] Webserver started at http://127.0.0.1:43707/ using document root <none> and password file <none>
I20260808 04:09:15.892738 4042 fs_manager.cc:359] Metadata directory not provided
I20260808 04:09:15.892856 4042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:09:15.893142 4042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:09:15.896034 4042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786161826605655-5452-0/minicluster-data/master-0/data/instance:
uuid: "aab44f036f1c465385496d7476298203"
format_stamp: "Formatted at 2026-08-08 04:09:15 on dist-test-slave-1l3l"
server_key: "cd6fcc231195e2da767f1aa0de6dad15"
server_key_iv: "b54286edfa4c7b076311e09c52d12f1a"
server_key_version: "encryptionkey@0"
I20260808 04:09:15.896739 4042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786161826605655-5452-0/minicluster-data/master-0/wal/instance:
uuid: "aab44f036f1c465385496d7476298203"
format_stamp: "Formatted at 2026-08-08 04:09:15 on dist-test-slave-1l3l"
server_key: "cd6fcc231195e2da767f1aa0de6dad15"
server_key_iv: "b54286edfa4c7b076311e09c52d12f1a"
server_key_version: "encryptionkey@0"
I20260808 04:09:15.901463 4042 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 04:09:15.904649 4061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:09:15.905869 4042 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 04:09:15.906032 4042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786161826605655-5452-0/minicluster-data/master-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786161826605655-5452-0/minicluster-data/master-0/wal
uuid: "aab44f036f1c465385496d7476298203"
format_stamp: "Formatted at 2026-08-08 04:09:15 on dist-test-slave-1l3l"
server_key: "cd6fcc231195e2da767f1aa0de6dad15"
server_key_iv: "b54286edfa4c7b076311e09c52d12f1a"
server_key_version: "encryptionkey@0"
I20260808 04:09:15.906203 4042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786161826605655-5452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786161826605655-5452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786161826605655-5452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:09:15.921976 4042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:09:15.925464 4042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:09:15.925705 4042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:09:15.941835 4042 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36067
I20260808 04:09:15.941869 4132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36067 every 8 connection(s)
I20260808 04:09:15.943074 4042 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
I20260808 04:09:15.946453 4133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:09:15.949352 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 4042
I20260808 04:09:15.949542 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786161826605655-5452-0/minicluster-data/master-0/wal/instance
I20260808 04:09:15.950075 5452 external_mini_cluster.cc:1428] Setting key e745e6093bbfc8f05c55308af447873f
I20260808 04:09:15.957604 4133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aab44f036f1c465385496d7476298203: Bootstrap starting.
I20260808 04:09:15.960870 4133 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aab44f036f1c465385496d7476298203: Neither blocks nor log segments found. Creating new log.
I20260808 04:09:15.961964 4133 log.cc:824] T 00000000000000000000000000000000 P aab44f036f1c465385496d7476298203: Log is configured to *not* fsync() on all Append() calls
I20260808 04:09:15.962504 4137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:15.951698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.111:59307 (local address 127.0.0.1:36067)
0808 04:09:15.952433 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:15.952444 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:15.952484 (+ 40us) server_negotiation.cc:407] Connection header received
0808 04:09:15.954070 (+ 1586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:15.954089 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:15.954457 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:15.954755 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:15.955286 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:15.956603 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:15.958335 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:15.958978 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:15.959100 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:15.960305 (+ 1205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:09:15.960312 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:09:15.960324 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:09:15.960826 (+ 502us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:09:15.960909 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:09:15.960933 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:09:15.961076 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:15.961333 (+ 257us) server_negotiation.cc:299] Negotiation successful
0808 04:09:15.961503 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":316,"threads_started":1}
I20260808 04:09:15.964793 4133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aab44f036f1c465385496d7476298203: No bootstrap required, opened a new log
I20260808 04:09:15.966109 5452 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:09:15.972318 4133 raft_consensus.cc:348] T 00000000000000000000000000000000 P aab44f036f1c465385496d7476298203 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aab44f036f1c465385496d7476298203" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36067 } }
I20260808 04:09:15.972642 4133 raft_consensus.cc:374] T 00000000000000000000000000000000 P aab44f036f1c465385496d7476298203 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:09:15.972761 4133 raft_consensus.cc:729] T 00000000000000000000000000000000 P aab44f036f1c465385496d7476298203 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aab44f036f1c465385496d7476298203, State: Initialized, Role: FOLLOWER
I20260808 04:09:15.973377 4133 consensus_queue.cc:260] T 00000000000000000000000000000000 P aab44f036f1c465385496d7476298203 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aab44f036f1c465385496d7476298203" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36067 } }
I20260808 04:09:15.973646 4133 raft_consensus.cc:388] T 00000000000000000000000000000000 P aab44f036f1c465385496d7476298203 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:09:15.973773 4133 raft_consensus.cc:482] T 00000000000000000000000000000000 P aab44f036f1c465385496d7476298203 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:09:15.973937 4133 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aab44f036f1c465385496d7476298203 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:09:15.976670 4137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:15.970358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.111:32843 (local address 127.0.0.1:36067)
0808 04:09:15.970529 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:15.970538 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:15.970883 (+ 345us) server_negotiation.cc:407] Connection header received
0808 04:09:15.971065 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:15.971069 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:15.971129 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:15.971203 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:15.971810 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:15.972528 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:15.974324 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:15.974775 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:15.974870 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:15.976024 (+ 1154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:09:15.976029 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:09:15.976032 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:09:15.976047 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:09:15.976099 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:09:15.976114 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:09:15.976192 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:15.976346 (+ 154us) server_negotiation.cc:299] Negotiation successful
0808 04:09:15.976384 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 04:09:15.976779 4133 raft_consensus.cc:504] T 00000000000000000000000000000000 P aab44f036f1c465385496d7476298203 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aab44f036f1c465385496d7476298203" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36067 } }
I20260808 04:09:15.977315 4133 leader_election.cc:302] T 00000000000000000000000000000000 P aab44f036f1c465385496d7476298203 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aab44f036f1c465385496d7476298203; no voters:
I20260808 04:09:15.977854 4133 leader_election.cc:288] T 00000000000000000000000000000000 P aab44f036f1c465385496d7476298203 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:09:15.977996 4145 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aab44f036f1c465385496d7476298203 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:09:15.978667 4145 raft_consensus.cc:686] T 00000000000000000000000000000000 P aab44f036f1c465385496d7476298203 [term 1 LEADER]: Becoming Leader. State: Replica: aab44f036f1c465385496d7476298203, State: Running, Role: LEADER
I20260808 04:09:15.979218 4145 consensus_queue.cc:237] T 00000000000000000000000000000000 P aab44f036f1c465385496d7476298203 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aab44f036f1c465385496d7476298203" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36067 } }
I20260808 04:09:15.979857 4133 sys_catalog.cc:564] T 00000000000000000000000000000000 P aab44f036f1c465385496d7476298203 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:09:15.981951 4147 sys_catalog.cc:455] T 00000000000000000000000000000000 P aab44f036f1c465385496d7476298203 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aab44f036f1c465385496d7476298203. Latest consensus state: current_term: 1 leader_uuid: "aab44f036f1c465385496d7476298203" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aab44f036f1c465385496d7476298203" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36067 } } }
I20260808 04:09:15.982314 4147 sys_catalog.cc:458] T 00000000000000000000000000000000 P aab44f036f1c465385496d7476298203 [sys.catalog]: This master's current role is: LEADER
I20260808 04:09:15.984992 4155 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:09:15.984833 4146 sys_catalog.cc:455] T 00000000000000000000000000000000 P aab44f036f1c465385496d7476298203 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aab44f036f1c465385496d7476298203" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aab44f036f1c465385496d7476298203" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36067 } } }
I20260808 04:09:15.985172 4146 sys_catalog.cc:458] T 00000000000000000000000000000000 P aab44f036f1c465385496d7476298203 [sys.catalog]: This master's current role is: LEADER
I20260808 04:09:15.989825 4155 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:09:15.998515 4155 catalog_manager.cc:1347] Generated new cluster ID: 2ad16e29d3f54b21a06dc2a59bb0d8b7
I20260808 04:09:15.998607 4155 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:09:16.012787 4155 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:09:16.013969 4155 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:09:16.024758 4155 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aab44f036f1c465385496d7476298203: Generated new TSK 0
I20260808 04:09:16.025627 4155 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:09:16.035768 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 4042
W20260808 04:09:16.037000 4127 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.111:59307
W20260808 04:09:16.037858 5452 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 4042: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 04:09:16.038019 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 4042
W20260808 04:09:16.060508 4014 connection.cc:535] client connection to 127.0.0.1:36067 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36067): Connection reset by peer (error 104)
2026-08-08T04:09:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1973 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786161826605655-5452-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:09:16 dist-test-slave-1l3l krb5kdc[4178](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:09:16 dist-test-slave-1l3l krb5kdc[4178](info): set up 2 sockets
Aug 08 04:09:16 dist-test-slave-1l3l krb5kdc[4178](info): commencing operation
krb5kdc: starting...
W20260808 04:09:17.235787 5452 mini_kdc.cc:121] Time spent starting KDC: real 1.156s user 0.011s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786161826605655-5452-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786161826605655-5452-0/krb5kdc/test-user.keytab.
Aug 08 04:09:17 dist-test-slave-1l3l krb5kdc[4178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162157, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:09:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:09:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786161826605655-5452-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786161826605655-5452-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 04:09:17.512552 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786161826605655-5452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786161826605655-5452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786161826605655-5452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33461
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786161826605655-5452-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33461
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:09:17.834043 4196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:09:17.834318 4196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:09:17.834467 4196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:09:17.841532 4196 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:09:17.841635 4196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:09:17.841707 4196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:09:17.841785 4196 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:09:17.841845 4196 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:09:17.851452 4196 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786161826605655-5452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786161826605655-5452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33461
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786161826605655-5452-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33461
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786161826605655-5452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:09:17.852726 4196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:09:17.854045 4196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:09:17.865028 4202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:09:17.865061 4206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:09:17.865105 4201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:09:17.865909 4196 server_base.cc:1034] running on GCE node
I20260808 04:09:17.866650 4196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:09:17.867934 4196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:09:17.869145 4196 hybrid_clock.cc:648] HybridClock initialized: now 1786162157869102 us; error 34 us; skew 500 ppm
I20260808 04:09:17.869498 4196 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:09:17 dist-test-slave-1l3l krb5kdc[4178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162157, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:09:17.876865 4196 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 04:09:17.878293 4196 webserver.cc:460] Webserver started at http://127.0.0.1:39611/ using document root <none> and password file <none>
I20260808 04:09:17.878928 4196 fs_manager.cc:359] Metadata directory not provided
I20260808 04:09:17.879045 4196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:09:17.879331 4196 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:09:17.882212 4196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786161826605655-5452-0/minicluster-data/master-0/data/instance:
uuid: "b1f5c0a974bc4d99b0f3773338195762"
format_stamp: "Formatted at 2026-08-08 04:09:17 on dist-test-slave-1l3l"
server_key: "13b3492d985f058d5aa4cc340c540838"
server_key_iv: "d972dd4b50b8a42d3d3017238c6561f7"
server_key_version: "encryptionkey@0"
I20260808 04:09:17.882946 4196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786161826605655-5452-0/minicluster-data/master-0/wal/instance:
uuid: "b1f5c0a974bc4d99b0f3773338195762"
format_stamp: "Formatted at 2026-08-08 04:09:17 on dist-test-slave-1l3l"
server_key: "13b3492d985f058d5aa4cc340c540838"
server_key_iv: "d972dd4b50b8a42d3d3017238c6561f7"
server_key_version: "encryptionkey@0"
I20260808 04:09:17.887876 4196 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260808 04:09:17.891181 4215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:09:17.892545 4196 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:09:17.892706 4196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786161826605655-5452-0/minicluster-data/master-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786161826605655-5452-0/minicluster-data/master-0/wal
uuid: "b1f5c0a974bc4d99b0f3773338195762"
format_stamp: "Formatted at 2026-08-08 04:09:17 on dist-test-slave-1l3l"
server_key: "13b3492d985f058d5aa4cc340c540838"
server_key_iv: "d972dd4b50b8a42d3d3017238c6561f7"
server_key_version: "encryptionkey@0"
I20260808 04:09:17.892876 4196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786161826605655-5452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786161826605655-5452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786161826605655-5452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:09:17.898407 4196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:09:17.902395 4196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:09:17.902669 4196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:09:17.919282 4196 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33461
I20260808 04:09:17.919324 4295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33461 every 8 connection(s)
I20260808 04:09:17.920578 4196 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
I20260808 04:09:17.924093 4296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:09:17.927294 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 4196
I20260808 04:09:17.927486 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786161826605655-5452-0/minicluster-data/master-0/wal/instance
I20260808 04:09:17.928042 5452 external_mini_cluster.cc:1428] Setting key 39996307b2752fa7708ee61e267e2212
I20260808 04:09:17.938138 4300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:17.930229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.111:52643 (local address 127.0.0.1:33461)
0808 04:09:17.931119 (+ 890us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:17.931139 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:17.931197 (+ 58us) server_negotiation.cc:407] Connection header received
0808 04:09:17.933812 (+ 2615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:17.933853 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:17.934341 (+ 488us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:17.934771 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:17.936089 (+ 1318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:09:17.936098 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:09:17.936115 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:09:17.936134 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:09:17.936213 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:09:17.936249 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:09:17.936479 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:17.936700 (+ 221us) server_negotiation.cc:299] Negotiation successful
0808 04:09:17.936894 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":404,"threads_started":1}
I20260808 04:09:17.938299 4296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1f5c0a974bc4d99b0f3773338195762: Bootstrap starting.
I20260808 04:09:17.942126 5452 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:09:17.942214 4296 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b1f5c0a974bc4d99b0f3773338195762: Neither blocks nor log segments found. Creating new log.
I20260808 04:09:17.943645 4296 log.cc:824] T 00000000000000000000000000000000 P b1f5c0a974bc4d99b0f3773338195762: Log is configured to *not* fsync() on all Append() calls
I20260808 04:09:17.947656 4296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1f5c0a974bc4d99b0f3773338195762: No bootstrap required, opened a new log
I20260808 04:09:17.954267 4300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:17.947678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.111:51895 (local address 127.0.0.1:33461)
0808 04:09:17.947918 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:17.947925 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:17.952382 (+ 4457us) server_negotiation.cc:407] Connection header received
0808 04:09:17.952447 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:17.952451 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:17.952515 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:17.952629 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:17.953560 (+ 931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:09:17.953566 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:09:17.953570 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:09:17.953588 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:09:17.953638 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:09:17.953657 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:09:17.953777 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:17.953903 (+ 126us) server_negotiation.cc:299] Negotiation successful
0808 04:09:17.953968 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260808 04:09:17.957422 4296 raft_consensus.cc:348] T 00000000000000000000000000000000 P b1f5c0a974bc4d99b0f3773338195762 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1f5c0a974bc4d99b0f3773338195762" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33461 } }
I20260808 04:09:17.957762 4296 raft_consensus.cc:374] T 00000000000000000000000000000000 P b1f5c0a974bc4d99b0f3773338195762 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:09:17.957852 4296 raft_consensus.cc:729] T 00000000000000000000000000000000 P b1f5c0a974bc4d99b0f3773338195762 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1f5c0a974bc4d99b0f3773338195762, State: Initialized, Role: FOLLOWER
I20260808 04:09:17.958510 4296 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1f5c0a974bc4d99b0f3773338195762 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1f5c0a974bc4d99b0f3773338195762" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33461 } }
I20260808 04:09:17.958803 4296 raft_consensus.cc:388] T 00000000000000000000000000000000 P b1f5c0a974bc4d99b0f3773338195762 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:09:17.958956 4296 raft_consensus.cc:482] T 00000000000000000000000000000000 P b1f5c0a974bc4d99b0f3773338195762 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:09:17.959134 4296 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b1f5c0a974bc4d99b0f3773338195762 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:09:17.961573 4296 raft_consensus.cc:504] T 00000000000000000000000000000000 P b1f5c0a974bc4d99b0f3773338195762 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1f5c0a974bc4d99b0f3773338195762" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33461 } }
I20260808 04:09:17.962153 4296 leader_election.cc:302] T 00000000000000000000000000000000 P b1f5c0a974bc4d99b0f3773338195762 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1f5c0a974bc4d99b0f3773338195762; no voters:
I20260808 04:09:17.962810 4296 leader_election.cc:288] T 00000000000000000000000000000000 P b1f5c0a974bc4d99b0f3773338195762 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:09:17.962963 4311 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b1f5c0a974bc4d99b0f3773338195762 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:09:17.963586 4311 raft_consensus.cc:686] T 00000000000000000000000000000000 P b1f5c0a974bc4d99b0f3773338195762 [term 1 LEADER]: Becoming Leader. State: Replica: b1f5c0a974bc4d99b0f3773338195762, State: Running, Role: LEADER
I20260808 04:09:17.964083 4311 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1f5c0a974bc4d99b0f3773338195762 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1f5c0a974bc4d99b0f3773338195762" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33461 } }
I20260808 04:09:17.964948 4296 sys_catalog.cc:564] T 00000000000000000000000000000000 P b1f5c0a974bc4d99b0f3773338195762 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:09:17.966542 4312 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1f5c0a974bc4d99b0f3773338195762 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b1f5c0a974bc4d99b0f3773338195762" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1f5c0a974bc4d99b0f3773338195762" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33461 } } }
I20260808 04:09:17.966595 4313 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1f5c0a974bc4d99b0f3773338195762 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1f5c0a974bc4d99b0f3773338195762. Latest consensus state: current_term: 1 leader_uuid: "b1f5c0a974bc4d99b0f3773338195762" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1f5c0a974bc4d99b0f3773338195762" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33461 } } }
I20260808 04:09:17.966970 4312 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1f5c0a974bc4d99b0f3773338195762 [sys.catalog]: This master's current role is: LEADER
I20260808 04:09:17.967852 4313 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1f5c0a974bc4d99b0f3773338195762 [sys.catalog]: This master's current role is: LEADER
I20260808 04:09:17.968937 4318 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:09:17.974320 4318 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:09:17.983664 4318 catalog_manager.cc:1347] Generated new cluster ID: ca4edb0ce6084f1b908de6960204884b
I20260808 04:09:17.983798 4318 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:09:17.996266 4318 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:09:17.996716 4318 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:09:18.009541 4318 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b1f5c0a974bc4d99b0f3773338195762: Generated new TSK 0
I20260808 04:09:18.010313 4318 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:09:18.021719 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 4196
W20260808 04:09:18.022710 4291 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.111:52643
W20260808 04:09:18.023339 5452 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 4196: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 04:09:18.023474 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 4196
2026-08-08T04:09:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1983 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786161826605655-5452-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:09:18 dist-test-slave-1l3l krb5kdc[4344](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:09:18 dist-test-slave-1l3l krb5kdc[4344](info): set up 2 sockets
Aug 08 04:09:18 dist-test-slave-1l3l krb5kdc[4344](info): commencing operation
krb5kdc: starting...
W20260808 04:09:19.221908 5452 mini_kdc.cc:121] Time spent starting KDC: real 1.159s user 0.000s sys 0.039s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786161826605655-5452-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786161826605655-5452-0/krb5kdc/test-user.keytab.
Aug 08 04:09:19 dist-test-slave-1l3l krb5kdc[4344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:09:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:09:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786161826605655-5452-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786161826605655-5452-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 04:09:19.492753 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786161826605655-5452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786161826605655-5452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786161826605655-5452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39509
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786161826605655-5452-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39509
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:09:19.810436 4362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:09:19.810711 4362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:09:19.810861 4362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:09:19.817788 4362 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:09:19.817898 4362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:09:19.817975 4362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:09:19.818068 4362 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:09:19.818130 4362 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:09:19.827356 4362 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786161826605655-5452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786161826605655-5452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39509
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786161826605655-5452-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786161826605655-5452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:09:19.828547 4362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:09:19.829787 4362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:09:19.841358 4369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:09:19.841421 4371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:09:19.841849 4368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:09:19.842274 4362 server_base.cc:1034] running on GCE node
I20260808 04:09:19.842999 4362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:09:19.844369 4362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:09:19.845587 4362 hybrid_clock.cc:648] HybridClock initialized: now 1786162159845533 us; error 46 us; skew 500 ppm
I20260808 04:09:19.845916 4362 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:09:19 dist-test-slave-1l3l krb5kdc[4344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162159, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:09:19.848891 4362 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 04:09:19.850288 4362 webserver.cc:460] Webserver started at http://127.0.0.1:36475/ using document root <none> and password file <none>
I20260808 04:09:19.850937 4362 fs_manager.cc:359] Metadata directory not provided
I20260808 04:09:19.851048 4362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:09:19.851332 4362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:09:19.854167 4362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786161826605655-5452-0/minicluster-data/master-0/data/instance:
uuid: "00defc26397846c08ec8701724856719"
format_stamp: "Formatted at 2026-08-08 04:09:19 on dist-test-slave-1l3l"
server_key: "f23690074a39661ba1285d22c07e04a0"
server_key_iv: "5b9ba425bc6f81e61ea8d315f500bd53"
server_key_version: "encryptionkey@0"
I20260808 04:09:19.854934 4362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786161826605655-5452-0/minicluster-data/master-0/wal/instance:
uuid: "00defc26397846c08ec8701724856719"
format_stamp: "Formatted at 2026-08-08 04:09:19 on dist-test-slave-1l3l"
server_key: "f23690074a39661ba1285d22c07e04a0"
server_key_iv: "5b9ba425bc6f81e61ea8d315f500bd53"
server_key_version: "encryptionkey@0"
I20260808 04:09:19.859969 4362 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260808 04:09:19.863210 4379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:09:19.864517 4362 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:09:19.864672 4362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786161826605655-5452-0/minicluster-data/master-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786161826605655-5452-0/minicluster-data/master-0/wal
uuid: "00defc26397846c08ec8701724856719"
format_stamp: "Formatted at 2026-08-08 04:09:19 on dist-test-slave-1l3l"
server_key: "f23690074a39661ba1285d22c07e04a0"
server_key_iv: "5b9ba425bc6f81e61ea8d315f500bd53"
server_key_version: "encryptionkey@0"
I20260808 04:09:19.864847 4362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786161826605655-5452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786161826605655-5452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786161826605655-5452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:09:19.904012 4362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:09:19.908221 4362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:09:19.908490 4362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:09:19.926229 4362 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39509
I20260808 04:09:19.926293 4454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39509 every 8 connection(s)
I20260808 04:09:19.927591 4362 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
I20260808 04:09:19.932300 4455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:09:19.935931 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 4362
I20260808 04:09:19.936102 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786161826605655-5452-0/minicluster-data/master-0/wal/instance
I20260808 04:09:19.936556 5452 external_mini_cluster.cc:1428] Setting key d81cba2d60134c318b027708ea542e8a
I20260808 04:09:19.946873 4455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00defc26397846c08ec8701724856719: Bootstrap starting.
I20260808 04:09:19.950359 4455 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00defc26397846c08ec8701724856719: Neither blocks nor log segments found. Creating new log.
I20260808 04:09:19.951623 4455 log.cc:824] T 00000000000000000000000000000000 P 00defc26397846c08ec8701724856719: Log is configured to *not* fsync() on all Append() calls
I20260808 04:09:19.954523 4455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00defc26397846c08ec8701724856719: No bootstrap required, opened a new log
Aug 08 04:09:19 dist-test-slave-1l3l krb5kdc[4344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260808 04:09:19.963184 4455 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00defc26397846c08ec8701724856719 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00defc26397846c08ec8701724856719" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39509 } }
I20260808 04:09:19.963541 4455 raft_consensus.cc:374] T 00000000000000000000000000000000 P 00defc26397846c08ec8701724856719 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:09:19.963665 4455 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00defc26397846c08ec8701724856719 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00defc26397846c08ec8701724856719, State: Initialized, Role: FOLLOWER
I20260808 04:09:19.964263 4455 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00defc26397846c08ec8701724856719 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00defc26397846c08ec8701724856719" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39509 } }
I20260808 04:09:19.964557 4455 raft_consensus.cc:388] T 00000000000000000000000000000000 P 00defc26397846c08ec8701724856719 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:09:19.964704 4455 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00defc26397846c08ec8701724856719 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:09:19.964887 4455 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00defc26397846c08ec8701724856719 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:09:19.967525 4461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:19.938437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.111:57525 (local address 127.0.0.1:39509)
0808 04:09:19.945373 (+ 6936us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:19.945397 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:19.945777 (+ 380us) server_negotiation.cc:407] Connection header received
0808 04:09:19.948146 (+ 2369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:19.948172 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:19.948675 (+ 503us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:19.949092 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:19.950368 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:19.952374 (+ 2006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:19.953742 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:19.954722 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:19.954891 (+ 169us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:19.958736 (+ 3845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:09:19.958745 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:09:19.958763 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:09:19.958815 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:09:19.962722 (+ 3907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:09:19.963500 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:09:19.963509 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:09:19.963517 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:09:19.963821 (+ 304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:09:19.964342 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:09:19.964347 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:09:19.964350 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:09:19.965115 (+ 765us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:09:19.965432 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:19.965857 (+ 425us) server_negotiation.cc:299] Negotiation successful
0808 04:09:19.966190 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6702,"thread_start_us":6457,"threads_started":1}
I20260808 04:09:19.968111 4455 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00defc26397846c08ec8701724856719 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00defc26397846c08ec8701724856719" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39509 } }
I20260808 04:09:19.968577 4455 leader_election.cc:302] T 00000000000000000000000000000000 P 00defc26397846c08ec8701724856719 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 00defc26397846c08ec8701724856719; no voters:
I20260808 04:09:19.969192 4455 leader_election.cc:288] T 00000000000000000000000000000000 P 00defc26397846c08ec8701724856719 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:09:19.969296 4464 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00defc26397846c08ec8701724856719 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:09:19.971097 4455 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00defc26397846c08ec8701724856719 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:09:19.972052 5452 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:09:19.975992 4464 raft_consensus.cc:686] T 00000000000000000000000000000000 P 00defc26397846c08ec8701724856719 [term 1 LEADER]: Becoming Leader. State: Replica: 00defc26397846c08ec8701724856719, State: Running, Role: LEADER
I20260808 04:09:19.976629 4464 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00defc26397846c08ec8701724856719 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00defc26397846c08ec8701724856719" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39509 } }
I20260808 04:09:19.985321 4466 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00defc26397846c08ec8701724856719 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00defc26397846c08ec8701724856719. Latest consensus state: current_term: 1 leader_uuid: "00defc26397846c08ec8701724856719" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00defc26397846c08ec8701724856719" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39509 } } }
I20260808 04:09:19.986126 4466 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00defc26397846c08ec8701724856719 [sys.catalog]: This master's current role is: LEADER
I20260808 04:09:19.985845 4465 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00defc26397846c08ec8701724856719 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "00defc26397846c08ec8701724856719" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00defc26397846c08ec8701724856719" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39509 } } }
I20260808 04:09:19.986589 4465 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00defc26397846c08ec8701724856719 [sys.catalog]: This master's current role is: LEADER
I20260808 04:09:19.988194 4484 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:09:19.993520 4484 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:09:20.010453 4484 catalog_manager.cc:1347] Generated new cluster ID: b34bebb7fb9a487e95a72f4ad79a3838
I20260808 04:09:20.010553 4484 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:09:20.017889 4461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:19.989667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.111:41685 (local address 127.0.0.1:39509)
0808 04:09:19.989826 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:19.989832 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:20.000619 (+ 10787us) server_negotiation.cc:407] Connection header received
0808 04:09:20.000937 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:20.000944 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:20.001031 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:20.001110 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:20.003268 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:20.004560 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:20.006578 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:20.007313 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:20.007411 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:20.012225 (+ 4814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:09:20.012229 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:09:20.012232 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:09:20.012262 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:09:20.015023 (+ 2761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:09:20.015796 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:09:20.015801 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:09:20.015804 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:09:20.015949 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:09:20.016381 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:09:20.016385 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:09:20.016387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:09:20.016707 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:09:20.016974 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:20.017359 (+ 385us) server_negotiation.cc:299] Negotiation successful
0808 04:09:20.017524 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260808 04:09:20.027980 4484 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:09:20.029663 4484 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:09:20.045626 4484 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 00defc26397846c08ec8701724856719: Generated new TSK 0
I20260808 04:09:20.046514 4484 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:09:20.055243 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 4362
I20260808 04:09:20.056453 4449 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:09:20.153075 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 4362
W20260808 04:09:20.173395 4340 connection.cc:535] client connection to 127.0.0.1:39509 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39509): Connection reset by peer (error 104)
2026-08-08T04:09:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2133 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786161826605655-5452-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:09:20 dist-test-slave-1l3l krb5kdc[4497](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:09:20 dist-test-slave-1l3l krb5kdc[4497](info): set up 2 sockets
Aug 08 04:09:20 dist-test-slave-1l3l krb5kdc[4497](info): commencing operation
krb5kdc: starting...
W20260808 04:09:21.357806 5452 mini_kdc.cc:121] Time spent starting KDC: real 1.162s user 0.006s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786161826605655-5452-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786161826605655-5452-0/krb5kdc/test-user.keytab.
Aug 08 04:09:21 dist-test-slave-1l3l krb5kdc[4497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:09:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:09:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786161826605655-5452-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786161826605655-5452-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 04:09:21.634902 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786161826605655-5452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786161826605655-5452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786161826605655-5452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35955
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786161826605655-5452-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35955
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:09:21.959578 4515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:09:21.959872 4515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:09:21.960012 4515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:09:21.966948 4515 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:09:21.967067 4515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:09:21.967146 4515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:09:21.967231 4515 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:09:21.967294 4515 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:09:21.976737 4515 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786161826605655-5452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786161826605655-5452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35955
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786161826605655-5452-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786161826605655-5452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:09:21.977936 4515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:09:21.979176 4515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:09:21.989976 4520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:09:21.990085 4521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:09:21.990118 4523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:09:21.991092 4515 server_base.cc:1034] running on GCE node
I20260808 04:09:21.991879 4515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:09:21.993108 4515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:09:21.994303 4515 hybrid_clock.cc:648] HybridClock initialized: now 1786162161994256 us; error 42 us; skew 500 ppm
I20260808 04:09:21.994611 4515 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:09:21 dist-test-slave-1l3l krb5kdc[4497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162161, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:09:21.997534 4515 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 04:09:21.998889 4515 webserver.cc:460] Webserver started at http://127.0.0.1:35385/ using document root <none> and password file <none>
I20260808 04:09:21.999483 4515 fs_manager.cc:359] Metadata directory not provided
I20260808 04:09:21.999594 4515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:09:21.999912 4515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:09:22.002652 4515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786161826605655-5452-0/minicluster-data/master-0/data/instance:
uuid: "14739a3b01534594b8897faafbc46ad5"
format_stamp: "Formatted at 2026-08-08 04:09:21 on dist-test-slave-1l3l"
server_key: "0d25ce6b846a479854f65e3bd6119636"
server_key_iv: "5c9a7993a9eee209b71129f482ed3a1d"
server_key_version: "encryptionkey@0"
I20260808 04:09:22.003383 4515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786161826605655-5452-0/minicluster-data/master-0/wal/instance:
uuid: "14739a3b01534594b8897faafbc46ad5"
format_stamp: "Formatted at 2026-08-08 04:09:21 on dist-test-slave-1l3l"
server_key: "0d25ce6b846a479854f65e3bd6119636"
server_key_iv: "5c9a7993a9eee209b71129f482ed3a1d"
server_key_version: "encryptionkey@0"
I20260808 04:09:22.008175 4515 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260808 04:09:22.011343 4533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:09:22.012570 4515 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:09:22.012725 4515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786161826605655-5452-0/minicluster-data/master-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786161826605655-5452-0/minicluster-data/master-0/wal
uuid: "14739a3b01534594b8897faafbc46ad5"
format_stamp: "Formatted at 2026-08-08 04:09:21 on dist-test-slave-1l3l"
server_key: "0d25ce6b846a479854f65e3bd6119636"
server_key_iv: "5c9a7993a9eee209b71129f482ed3a1d"
server_key_version: "encryptionkey@0"
I20260808 04:09:22.012898 4515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786161826605655-5452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786161826605655-5452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786161826605655-5452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:09:22.033573 4515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:09:22.037200 4515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:09:22.037439 4515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:09:22.054064 4515 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35955
I20260808 04:09:22.054095 4606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35955 every 8 connection(s)
I20260808 04:09:22.055284 4515 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
I20260808 04:09:22.058799 4607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:09:22.062129 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 4515
I20260808 04:09:22.062294 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786161826605655-5452-0/minicluster-data/master-0/wal/instance
I20260808 04:09:22.062723 5452 external_mini_cluster.cc:1428] Setting key 270fe441ae406db27edc7411fc3bbc1c
I20260808 04:09:22.071594 4607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14739a3b01534594b8897faafbc46ad5: Bootstrap starting.
I20260808 04:09:22.075192 4607 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14739a3b01534594b8897faafbc46ad5: Neither blocks nor log segments found. Creating new log.
I20260808 04:09:22.076498 4607 log.cc:824] T 00000000000000000000000000000000 P 14739a3b01534594b8897faafbc46ad5: Log is configured to *not* fsync() on all Append() calls
I20260808 04:09:22.077046 4611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:22.064508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.111:57203 (local address 127.0.0.1:35955)
0808 04:09:22.065336 (+ 828us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:22.065375 (+ 39us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:22.065426 (+ 51us) server_negotiation.cc:407] Connection header received
0808 04:09:22.067240 (+ 1814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:22.067266 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:22.067785 (+ 519us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:22.068235 (+ 450us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:22.068749 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:22.070761 (+ 2012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:22.072124 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:22.073096 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:22.073300 (+ 204us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:22.074329 (+ 1029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:09:22.074339 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:09:22.074356 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:09:22.075014 (+ 658us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:09:22.075108 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:09:22.075147 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:09:22.075323 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:22.075597 (+ 274us) server_negotiation.cc:299] Negotiation successful
0808 04:09:22.075797 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":376,"threads_started":1}
I20260808 04:09:22.079252 4607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14739a3b01534594b8897faafbc46ad5: No bootstrap required, opened a new log
I20260808 04:09:22.080531 5452 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:09:22.088678 4607 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14739a3b01534594b8897faafbc46ad5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14739a3b01534594b8897faafbc46ad5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35955 } }
I20260808 04:09:22.088991 4607 raft_consensus.cc:374] T 00000000000000000000000000000000 P 14739a3b01534594b8897faafbc46ad5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:09:22.089082 4607 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14739a3b01534594b8897faafbc46ad5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14739a3b01534594b8897faafbc46ad5, State: Initialized, Role: FOLLOWER
I20260808 04:09:22.089695 4607 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14739a3b01534594b8897faafbc46ad5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14739a3b01534594b8897faafbc46ad5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35955 } }
I20260808 04:09:22.089977 4607 raft_consensus.cc:388] T 00000000000000000000000000000000 P 14739a3b01534594b8897faafbc46ad5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:09:22.090133 4607 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14739a3b01534594b8897faafbc46ad5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:09:22.090306 4607 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14739a3b01534594b8897faafbc46ad5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:09:22.091795 4611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:22.085689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.111:33483 (local address 127.0.0.1:35955)
0808 04:09:22.085856 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:22.085861 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:22.086142 (+ 281us) server_negotiation.cc:407] Connection header received
0808 04:09:22.086270 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:22.086275 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:22.086349 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:22.086461 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:22.086871 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:22.088007 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:22.089279 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:22.089897 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:22.090037 (+ 140us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:22.091067 (+ 1030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:09:22.091074 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:09:22.091079 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:09:22.091099 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:09:22.091163 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:09:22.091190 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:09:22.091305 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:22.091408 (+ 103us) server_negotiation.cc:299] Negotiation successful
0808 04:09:22.091461 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 04:09:22.093123 4607 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14739a3b01534594b8897faafbc46ad5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14739a3b01534594b8897faafbc46ad5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35955 } }
I20260808 04:09:22.093565 4607 leader_election.cc:302] T 00000000000000000000000000000000 P 14739a3b01534594b8897faafbc46ad5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 14739a3b01534594b8897faafbc46ad5; no voters:
I20260808 04:09:22.094041 4607 leader_election.cc:288] T 00000000000000000000000000000000 P 14739a3b01534594b8897faafbc46ad5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:09:22.094151 4623 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 14739a3b01534594b8897faafbc46ad5 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:09:22.094765 4623 raft_consensus.cc:686] T 00000000000000000000000000000000 P 14739a3b01534594b8897faafbc46ad5 [term 1 LEADER]: Becoming Leader. State: Replica: 14739a3b01534594b8897faafbc46ad5, State: Running, Role: LEADER
I20260808 04:09:22.095261 4623 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14739a3b01534594b8897faafbc46ad5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14739a3b01534594b8897faafbc46ad5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35955 } }
I20260808 04:09:22.095989 4607 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14739a3b01534594b8897faafbc46ad5 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:09:22.097971 4626 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14739a3b01534594b8897faafbc46ad5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14739a3b01534594b8897faafbc46ad5. Latest consensus state: current_term: 1 leader_uuid: "14739a3b01534594b8897faafbc46ad5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14739a3b01534594b8897faafbc46ad5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35955 } } }
I20260808 04:09:22.097958 4625 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14739a3b01534594b8897faafbc46ad5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "14739a3b01534594b8897faafbc46ad5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14739a3b01534594b8897faafbc46ad5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35955 } } }
I20260808 04:09:22.098426 4626 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14739a3b01534594b8897faafbc46ad5 [sys.catalog]: This master's current role is: LEADER
I20260808 04:09:22.098426 4625 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14739a3b01534594b8897faafbc46ad5 [sys.catalog]: This master's current role is: LEADER
I20260808 04:09:22.104621 4633 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:09:22.109652 4633 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:09:22.118108 4633 catalog_manager.cc:1347] Generated new cluster ID: f7f42a9519ee43299a3d5216bb6946e6
I20260808 04:09:22.118199 4633 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:09:22.136057 4633 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:09:22.137233 4633 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:09:22.145314 4633 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 14739a3b01534594b8897faafbc46ad5: Generated new TSK 0
I20260808 04:09:22.146009 4633 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:09:22.160209 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 4515
W20260808 04:09:22.161288 4600 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.111:57203
W20260808 04:09:22.161989 5452 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 4515: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 04:09:22.162125 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 4515
W20260808 04:09:22.185295 4495 connection.cc:535] client connection to 127.0.0.1:35955 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35955): Connection reset by peer (error 104)
2026-08-08T04:09:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2007 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786161826605655-5452-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:09:22 dist-test-slave-1l3l krb5kdc[4655](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:09:22 dist-test-slave-1l3l krb5kdc[4655](info): set up 2 sockets
Aug 08 04:09:22 dist-test-slave-1l3l krb5kdc[4655](info): commencing operation
krb5kdc: starting...
W20260808 04:09:23.361040 5452 mini_kdc.cc:121] Time spent starting KDC: real 1.157s user 0.000s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786161826605655-5452-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786161826605655-5452-0/krb5kdc/test-user.keytab.
Aug 08 04:09:23 dist-test-slave-1l3l krb5kdc[4655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:09:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:09:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786161826605655-5452-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786161826605655-5452-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 04:09:23.623754 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786161826605655-5452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786161826605655-5452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786161826605655-5452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43539
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786161826605655-5452-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43539
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:09:23.944164 4674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:09:23.944425 4674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:09:23.944558 4674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:09:23.951462 4674 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:09:23.951560 4674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:09:23.951638 4674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:09:23.951733 4674 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:09:23.951795 4674 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:09:23.961035 4674 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786161826605655-5452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786161826605655-5452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43539
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786161826605655-5452-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786161826605655-5452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:09:23.962159 4674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:09:23.963362 4674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:09:23.974234 4679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:09:23.974298 4682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:09:23.974339 4680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:09:23.975220 4674 server_base.cc:1034] running on GCE node
I20260808 04:09:23.975975 4674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:09:23.977180 4674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:09:23.978400 4674 hybrid_clock.cc:648] HybridClock initialized: now 1786162163978341 us; error 46 us; skew 500 ppm
I20260808 04:09:23.978727 4674 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:09:23 dist-test-slave-1l3l krb5kdc[4655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162163, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:09:23.987547 4674 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 04:09:23.988940 4674 webserver.cc:460] Webserver started at http://127.0.0.1:45595/ using document root <none> and password file <none>
I20260808 04:09:23.989576 4674 fs_manager.cc:359] Metadata directory not provided
I20260808 04:09:23.989697 4674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:09:23.989995 4674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:09:23.992827 4674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786161826605655-5452-0/minicluster-data/master-0/data/instance:
uuid: "0e92131378424155a88078b57134b8f5"
format_stamp: "Formatted at 2026-08-08 04:09:23 on dist-test-slave-1l3l"
server_key: "ae80fa97618860cad8b09f2d8fa2c891"
server_key_iv: "995f6bab04d2050f02675b233301de68"
server_key_version: "encryptionkey@0"
I20260808 04:09:23.993577 4674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786161826605655-5452-0/minicluster-data/master-0/wal/instance:
uuid: "0e92131378424155a88078b57134b8f5"
format_stamp: "Formatted at 2026-08-08 04:09:23 on dist-test-slave-1l3l"
server_key: "ae80fa97618860cad8b09f2d8fa2c891"
server_key_iv: "995f6bab04d2050f02675b233301de68"
server_key_version: "encryptionkey@0"
I20260808 04:09:23.998459 4674 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260808 04:09:24.001684 4689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:09:24.002996 4674 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 04:09:24.003154 4674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786161826605655-5452-0/minicluster-data/master-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786161826605655-5452-0/minicluster-data/master-0/wal
uuid: "0e92131378424155a88078b57134b8f5"
format_stamp: "Formatted at 2026-08-08 04:09:23 on dist-test-slave-1l3l"
server_key: "ae80fa97618860cad8b09f2d8fa2c891"
server_key_iv: "995f6bab04d2050f02675b233301de68"
server_key_version: "encryptionkey@0"
I20260808 04:09:24.003327 4674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786161826605655-5452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786161826605655-5452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786161826605655-5452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:09:24.008675 4674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:09:24.012372 4674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:09:24.012629 4674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:09:24.029888 4674 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43539
I20260808 04:09:24.029928 4756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43539 every 8 connection(s)
I20260808 04:09:24.031229 4674 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
I20260808 04:09:24.034813 4757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:09:24.038542 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 4674
I20260808 04:09:24.038723 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786161826605655-5452-0/minicluster-data/master-0/wal/instance
I20260808 04:09:24.039252 5452 external_mini_cluster.cc:1428] Setting key 84aad0bd4ba24ae0f29ab507a588e2bb
I20260808 04:09:24.046383 4761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:24.040993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.111:51551 (local address 127.0.0.1:43539)
0808 04:09:24.041675 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:24.041686 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:24.041728 (+ 42us) server_negotiation.cc:407] Connection header received
0808 04:09:24.043267 (+ 1539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:24.043289 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:24.043609 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:24.043964 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:24.044975 (+ 1011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:09:24.044981 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:09:24.044992 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:09:24.045007 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:09:24.045064 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:09:24.045083 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:09:24.045244 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:24.045368 (+ 124us) server_negotiation.cc:299] Negotiation successful
0808 04:09:24.045525 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":306,"threads_started":1}
I20260808 04:09:24.046967 4757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e92131378424155a88078b57134b8f5: Bootstrap starting.
I20260808 04:09:24.049909 5452 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:09:24.050377 4757 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e92131378424155a88078b57134b8f5: Neither blocks nor log segments found. Creating new log.
I20260808 04:09:24.051559 4757 log.cc:824] T 00000000000000000000000000000000 P 0e92131378424155a88078b57134b8f5: Log is configured to *not* fsync() on all Append() calls
I20260808 04:09:24.054666 4757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e92131378424155a88078b57134b8f5: No bootstrap required, opened a new log
I20260808 04:09:24.057126 4761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:24.054801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.111:35337 (local address 127.0.0.1:43539)
0808 04:09:24.054964 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:24.054969 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:24.055348 (+ 379us) server_negotiation.cc:407] Connection header received
0808 04:09:24.055467 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:24.055470 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:24.055521 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:24.055597 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:24.056552 (+ 955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:09:24.056557 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:09:24.056560 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:09:24.056575 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:09:24.056613 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:09:24.056630 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:09:24.056719 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:24.056822 (+ 103us) server_negotiation.cc:299] Negotiation successful
0808 04:09:24.056868 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 04:09:24.062515 4757 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e92131378424155a88078b57134b8f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e92131378424155a88078b57134b8f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43539 } }
I20260808 04:09:24.062844 4757 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0e92131378424155a88078b57134b8f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:09:24.062944 4757 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e92131378424155a88078b57134b8f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e92131378424155a88078b57134b8f5, State: Initialized, Role: FOLLOWER
I20260808 04:09:24.063469 4757 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e92131378424155a88078b57134b8f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e92131378424155a88078b57134b8f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43539 } }
I20260808 04:09:24.063720 4757 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0e92131378424155a88078b57134b8f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:09:24.063853 4757 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e92131378424155a88078b57134b8f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:09:24.064023 4757 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e92131378424155a88078b57134b8f5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:09:24.066049 4757 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e92131378424155a88078b57134b8f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e92131378424155a88078b57134b8f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43539 } }
I20260808 04:09:24.066489 4757 leader_election.cc:302] T 00000000000000000000000000000000 P 0e92131378424155a88078b57134b8f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e92131378424155a88078b57134b8f5; no voters:
I20260808 04:09:24.067027 4757 leader_election.cc:288] T 00000000000000000000000000000000 P 0e92131378424155a88078b57134b8f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:09:24.067131 4769 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e92131378424155a88078b57134b8f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:09:24.067768 4769 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e92131378424155a88078b57134b8f5 [term 1 LEADER]: Becoming Leader. State: Replica: 0e92131378424155a88078b57134b8f5, State: Running, Role: LEADER
I20260808 04:09:24.068315 4769 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e92131378424155a88078b57134b8f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e92131378424155a88078b57134b8f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43539 } }
I20260808 04:09:24.068883 4757 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e92131378424155a88078b57134b8f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:09:24.071170 4770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e92131378424155a88078b57134b8f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0e92131378424155a88078b57134b8f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e92131378424155a88078b57134b8f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43539 } } }
I20260808 04:09:24.071604 4770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e92131378424155a88078b57134b8f5 [sys.catalog]: This master's current role is: LEADER
I20260808 04:09:24.072518 4771 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e92131378424155a88078b57134b8f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e92131378424155a88078b57134b8f5. Latest consensus state: current_term: 1 leader_uuid: "0e92131378424155a88078b57134b8f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e92131378424155a88078b57134b8f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43539 } } }
I20260808 04:09:24.072818 4771 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e92131378424155a88078b57134b8f5 [sys.catalog]: This master's current role is: LEADER
I20260808 04:09:24.073630 4778 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:09:24.078451 4778 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:09:24.086907 4778 catalog_manager.cc:1347] Generated new cluster ID: d04945207be54a309edadc63c2dffa0a
I20260808 04:09:24.087008 4778 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:09:24.101621 4778 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:09:24.101974 4778 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:09:24.109560 4778 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0e92131378424155a88078b57134b8f5: Generated new TSK 0
I20260808 04:09:24.110200 4778 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:09:24.114926 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 4674
W20260808 04:09:24.115919 4749 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.111:51551
W20260808 04:09:24.116575 5452 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 4674: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 04:09:24.116696 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 4674
2026-08-08T04:09:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1952 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786161826605655-5452-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:09:24 dist-test-slave-1l3l krb5kdc[4793](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:09:24 dist-test-slave-1l3l krb5kdc[4793](info): set up 2 sockets
Aug 08 04:09:24 dist-test-slave-1l3l krb5kdc[4793](info): commencing operation
krb5kdc: starting...
W20260808 04:09:25.327890 5452 mini_kdc.cc:121] Time spent starting KDC: real 1.172s user 0.000s sys 0.039s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786161826605655-5452-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786161826605655-5452-0/krb5kdc/test-user.keytab.
Aug 08 04:09:25 dist-test-slave-1l3l krb5kdc[4793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:09:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:09:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.83.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab.
Entry for principal kudu/127.5.83.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab.
I20260808 04:09:25.602829 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786161826605655-5452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786161826605655-5452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786161826605655-5452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:41761
--webserver_interface=127.5.83.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab
--principal=kudu/127.5.83.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:45539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.62:41761
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:09:25.921739 4813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:09:25.922019 4813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:09:25.922119 4813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:09:25.929026 4813 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:09:25.929123 4813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:09:25.929176 4813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:09:25.929235 4813 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:09:25.929277 4813 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:09:25.938479 4813 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:45539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786161826605655-5452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786161826605655-5452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.62:41761
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab
--principal=kudu/127.5.83.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:41761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786161826605655-5452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:09:25.939599 4813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:09:25.940790 4813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:09:25.951570 4818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:09:25.951686 4819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:09:25.951800 4821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:09:25.952589 4813 server_base.cc:1034] running on GCE node
I20260808 04:09:25.953325 4813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:09:25.954531 4813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:09:25.955749 4813 hybrid_clock.cc:648] HybridClock initialized: now 1786162165955692 us; error 27 us; skew 500 ppm
I20260808 04:09:25.956058 4813 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:09:25 dist-test-slave-1l3l krb5kdc[4793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162165, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:09:25.958865 4813 init.cc:377] Logged in from keytab as kudu/127.5.83.62@KRBTEST.COM (short username kudu)
I20260808 04:09:25.960253 4813 webserver.cc:460] Webserver started at http://127.5.83.62:41217/ using document root <none> and password file <none>
I20260808 04:09:25.960834 4813 fs_manager.cc:359] Metadata directory not provided
I20260808 04:09:25.960938 4813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:09:25.961177 4813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:09:25.963975 4813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786161826605655-5452-0/minicluster-data/master-0/data/instance:
uuid: "39b049de09d54cbe9760c888e1aa1c37"
format_stamp: "Formatted at 2026-08-08 04:09:25 on dist-test-slave-1l3l"
server_key: "126e283b16a1b8207e551d71d8e25605"
server_key_iv: "8459cf2b0922945157bcd932cb98aca9"
server_key_version: "encryptionkey@0"
I20260808 04:09:25.964646 4813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786161826605655-5452-0/minicluster-data/master-0/wal/instance:
uuid: "39b049de09d54cbe9760c888e1aa1c37"
format_stamp: "Formatted at 2026-08-08 04:09:25 on dist-test-slave-1l3l"
server_key: "126e283b16a1b8207e551d71d8e25605"
server_key_iv: "8459cf2b0922945157bcd932cb98aca9"
server_key_version: "encryptionkey@0"
I20260808 04:09:25.969410 4813 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 04:09:25.972970 4829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:09:25.974156 4813 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:09:25.974293 4813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786161826605655-5452-0/minicluster-data/master-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786161826605655-5452-0/minicluster-data/master-0/wal
uuid: "39b049de09d54cbe9760c888e1aa1c37"
format_stamp: "Formatted at 2026-08-08 04:09:25 on dist-test-slave-1l3l"
server_key: "126e283b16a1b8207e551d71d8e25605"
server_key_iv: "8459cf2b0922945157bcd932cb98aca9"
server_key_version: "encryptionkey@0"
I20260808 04:09:25.974452 4813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786161826605655-5452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786161826605655-5452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786161826605655-5452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:09:26.000032 4813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:09:26.003726 4813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:09:26.003961 4813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:09:26.020854 4813 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.62:41761
I20260808 04:09:26.020885 4894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.62:41761 every 8 connection(s)
I20260808 04:09:26.022176 4813 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
I20260808 04:09:26.025911 4895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:09:26.027891 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 4813
I20260808 04:09:26.028075 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786161826605655-5452-0/minicluster-data/master-0/wal/instance
I20260808 04:09:26.028635 5452 external_mini_cluster.cc:1428] Setting key 384402113c8b920a547f375bf2c87c2f
I20260808 04:09:26.037806 4895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39b049de09d54cbe9760c888e1aa1c37: Bootstrap starting.
I20260808 04:09:26.041435 4895 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 39b049de09d54cbe9760c888e1aa1c37: Neither blocks nor log segments found. Creating new log.
I20260808 04:09:26.042640 4895 log.cc:824] T 00000000000000000000000000000000 P 39b049de09d54cbe9760c888e1aa1c37: Log is configured to *not* fsync() on all Append() calls
Aug 08 04:09:26 dist-test-slave-1l3l krb5kdc[4793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:09:26.045368 4895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39b049de09d54cbe9760c888e1aa1c37: No bootstrap required, opened a new log
I20260808 04:09:26.051196 4899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:26.030410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48997 (local address 127.5.83.62:41761)
0808 04:09:26.031178 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:26.031190 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:26.031229 (+ 39us) server_negotiation.cc:407] Connection header received
0808 04:09:26.032888 (+ 1659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:26.032910 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:26.033275 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:26.033593 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:26.035362 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:26.036714 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:26.038474 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:26.039091 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:26.039222 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:26.044192 (+ 4970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:09:26.044200 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:09:26.044214 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:09:26.044252 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:09:26.047195 (+ 2943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:09:26.048028 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:09:26.048037 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:09:26.048043 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:09:26.048202 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:09:26.048650 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:09:26.048654 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:09:26.048656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:09:26.049153 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:09:26.049479 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:26.049964 (+ 485us) server_negotiation.cc:299] Negotiation successful
0808 04:09:26.050250 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":347,"threads_started":1}
I20260808 04:09:26.052911 4895 raft_consensus.cc:348] T 00000000000000000000000000000000 P 39b049de09d54cbe9760c888e1aa1c37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39b049de09d54cbe9760c888e1aa1c37" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 41761 } }
I20260808 04:09:26.053229 4895 raft_consensus.cc:374] T 00000000000000000000000000000000 P 39b049de09d54cbe9760c888e1aa1c37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:09:26.053339 4895 raft_consensus.cc:729] T 00000000000000000000000000000000 P 39b049de09d54cbe9760c888e1aa1c37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39b049de09d54cbe9760c888e1aa1c37, State: Initialized, Role: FOLLOWER
I20260808 04:09:26.053908 4895 consensus_queue.cc:260] T 00000000000000000000000000000000 P 39b049de09d54cbe9760c888e1aa1c37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39b049de09d54cbe9760c888e1aa1c37" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 41761 } }
I20260808 04:09:26.054162 4895 raft_consensus.cc:388] T 00000000000000000000000000000000 P 39b049de09d54cbe9760c888e1aa1c37 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:09:26.054287 4895 raft_consensus.cc:482] T 00000000000000000000000000000000 P 39b049de09d54cbe9760c888e1aa1c37 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:09:26.054450 4895 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 39b049de09d54cbe9760c888e1aa1c37 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:09:26.054773 5452 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:09:26.056635 4895 raft_consensus.cc:504] T 00000000000000000000000000000000 P 39b049de09d54cbe9760c888e1aa1c37 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39b049de09d54cbe9760c888e1aa1c37" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 41761 } }
I20260808 04:09:26.057083 4895 leader_election.cc:302] T 00000000000000000000000000000000 P 39b049de09d54cbe9760c888e1aa1c37 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 39b049de09d54cbe9760c888e1aa1c37; no voters:
I20260808 04:09:26.057667 4895 leader_election.cc:288] T 00000000000000000000000000000000 P 39b049de09d54cbe9760c888e1aa1c37 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:09:26.057744 4904 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 39b049de09d54cbe9760c888e1aa1c37 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:09:26.059788 4904 raft_consensus.cc:686] T 00000000000000000000000000000000 P 39b049de09d54cbe9760c888e1aa1c37 [term 1 LEADER]: Becoming Leader. State: Replica: 39b049de09d54cbe9760c888e1aa1c37, State: Running, Role: LEADER
I20260808 04:09:26.059954 4895 sys_catalog.cc:564] T 00000000000000000000000000000000 P 39b049de09d54cbe9760c888e1aa1c37 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:09:26.060359 4904 consensus_queue.cc:237] T 00000000000000000000000000000000 P 39b049de09d54cbe9760c888e1aa1c37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39b049de09d54cbe9760c888e1aa1c37" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 41761 } }
I20260808 04:09:26.063095 4910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39b049de09d54cbe9760c888e1aa1c37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 39b049de09d54cbe9760c888e1aa1c37. Latest consensus state: current_term: 1 leader_uuid: "39b049de09d54cbe9760c888e1aa1c37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39b049de09d54cbe9760c888e1aa1c37" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 41761 } } }
I20260808 04:09:26.063479 4910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39b049de09d54cbe9760c888e1aa1c37 [sys.catalog]: This master's current role is: LEADER
I20260808 04:09:26.064067 4908 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39b049de09d54cbe9760c888e1aa1c37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "39b049de09d54cbe9760c888e1aa1c37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39b049de09d54cbe9760c888e1aa1c37" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 41761 } } }
I20260808 04:09:26.064370 4916 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:09:26.064368 4908 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39b049de09d54cbe9760c888e1aa1c37 [sys.catalog]: This master's current role is: LEADER
I20260808 04:09:26.076282 4899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:26.058839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46039 (local address 127.5.83.62:41761)
0808 04:09:26.058997 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:26.059002 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:26.059516 (+ 514us) server_negotiation.cc:407] Connection header received
0808 04:09:26.059807 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:26.059813 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:26.059892 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:26.059968 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:26.061581 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:26.062691 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:26.064517 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:26.065239 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:26.065360 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:26.068741 (+ 3381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:09:26.068748 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:09:26.068754 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:09:26.068795 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:09:26.072679 (+ 3884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:09:26.073673 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:09:26.073679 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:09:26.073682 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:09:26.073816 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:09:26.074351 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:09:26.074357 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:09:26.074360 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:09:26.074771 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:09:26.075035 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:26.075501 (+ 466us) server_negotiation.cc:299] Negotiation successful
0808 04:09:26.075935 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 04:09:26.077139 4916 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:09:26.089262 4916 catalog_manager.cc:1347] Generated new cluster ID: 71b9532ff0174eff96058033b3d1b9f6
I20260808 04:09:26.089370 4916 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:09:26.114786 4916 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:09:26.116043 4916 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:09:26.122089 4916 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 39b049de09d54cbe9760c888e1aa1c37: Generated new TSK 0
I20260808 04:09:26.123004 4916 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:09:26.130045 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 4813
I20260808 04:09:26.131407 4887 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:09:26.219553 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 4813
W20260808 04:09:26.237306 4791 connection.cc:535] client connection to 127.5.83.62:41761 recv error: Network error: failed to read from TLS socket (remote: 127.5.83.62:41761): Connection reset by peer (error 104)
2026-08-08T04:09:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2099 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786161826605655-5452-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:09:26 dist-test-slave-1l3l krb5kdc[4935](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:09:26 dist-test-slave-1l3l krb5kdc[4935](info): set up 2 sockets
Aug 08 04:09:26 dist-test-slave-1l3l krb5kdc[4935](info): commencing operation
krb5kdc: starting...
W20260808 04:09:27.421183 5452 mini_kdc.cc:121] Time spent starting KDC: real 1.166s user 0.006s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786161826605655-5452-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786161826605655-5452-0/krb5kdc/test-user.keytab.
Aug 08 04:09:27 dist-test-slave-1l3l krb5kdc[4935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:09:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:09:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.83.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab.
Entry for principal kudu/127.5.83.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab.
I20260808 04:09:27.693001 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786161826605655-5452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786161826605655-5452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786161826605655-5452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:41953
--webserver_interface=127.5.83.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab
--principal=kudu/127.5.83.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:39673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.62:41953
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:09:28.019685 4952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:09:28.020063 4952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:09:28.020193 4952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:09:28.029747 4952 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:09:28.029865 4952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:09:28.029973 4952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:09:28.030057 4952 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:09:28.030118 4952 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:09:28.042879 4952 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:39673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786161826605655-5452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786161826605655-5452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.62:41953
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab
--principal=kudu/127.5.83.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:41953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786161826605655-5452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:09:28.044291 4952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:09:28.045575 4952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:09:28.056826 4959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:09:28.056866 4960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:09:28.057021 4962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:09:28.057704 4952 server_base.cc:1034] running on GCE node
I20260808 04:09:28.058408 4952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:09:28.059743 4952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:09:28.060951 4952 hybrid_clock.cc:648] HybridClock initialized: now 1786162168060906 us; error 38 us; skew 500 ppm
I20260808 04:09:28.061251 4952 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:09:28 dist-test-slave-1l3l krb5kdc[4935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162168, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:09:28.064203 4952 init.cc:377] Logged in from keytab as kudu/127.5.83.62@KRBTEST.COM (short username kudu)
I20260808 04:09:28.065491 4952 webserver.cc:460] Webserver started at http://127.5.83.62:40183/ using document root <none> and password file <none>
I20260808 04:09:28.066104 4952 fs_manager.cc:359] Metadata directory not provided
I20260808 04:09:28.066213 4952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:09:28.066488 4952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:09:28.069422 4952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786161826605655-5452-0/minicluster-data/master-0/data/instance:
uuid: "73971c7c2782457497b052fd1f54be46"
format_stamp: "Formatted at 2026-08-08 04:09:28 on dist-test-slave-1l3l"
server_key: "a6f5676cbfa7beac3ca660afd747a0f3"
server_key_iv: "d278b05e6bfbcb361fe57fd3585a3d5c"
server_key_version: "encryptionkey@0"
I20260808 04:09:28.070143 4952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786161826605655-5452-0/minicluster-data/master-0/wal/instance:
uuid: "73971c7c2782457497b052fd1f54be46"
format_stamp: "Formatted at 2026-08-08 04:09:28 on dist-test-slave-1l3l"
server_key: "a6f5676cbfa7beac3ca660afd747a0f3"
server_key_iv: "d278b05e6bfbcb361fe57fd3585a3d5c"
server_key_version: "encryptionkey@0"
I20260808 04:09:28.074841 4952 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260808 04:09:28.078078 4972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:09:28.079221 4952 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:09:28.079367 4952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786161826605655-5452-0/minicluster-data/master-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786161826605655-5452-0/minicluster-data/master-0/wal
uuid: "73971c7c2782457497b052fd1f54be46"
format_stamp: "Formatted at 2026-08-08 04:09:28 on dist-test-slave-1l3l"
server_key: "a6f5676cbfa7beac3ca660afd747a0f3"
server_key_iv: "d278b05e6bfbcb361fe57fd3585a3d5c"
server_key_version: "encryptionkey@0"
I20260808 04:09:28.079527 4952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786161826605655-5452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786161826605655-5452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786161826605655-5452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:09:28.102842 4952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:09:28.106812 4952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:09:28.107067 4952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:09:28.123386 4952 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.62:41953
I20260808 04:09:28.123420 5040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.62:41953 every 8 connection(s)
I20260808 04:09:28.124650 4952 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
I20260808 04:09:28.128121 5041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:09:28.128645 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 4952
I20260808 04:09:28.128818 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786161826605655-5452-0/minicluster-data/master-0/wal/instance
I20260808 04:09:28.129370 5452 external_mini_cluster.cc:1428] Setting key 8cdf4d46958d9486168c4a85fd6d8ad9
I20260808 04:09:28.140966 5041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73971c7c2782457497b052fd1f54be46: Bootstrap starting.
I20260808 04:09:28.142647 5044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:28.131157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57721 (local address 127.5.83.62:41953)
0808 04:09:28.131927 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:28.131938 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:28.131980 (+ 42us) server_negotiation.cc:407] Connection header received
0808 04:09:28.134039 (+ 2059us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:28.134059 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:28.134433 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:28.134733 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:28.135279 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:28.136596 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:28.138487 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:28.139129 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:28.139248 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:28.140509 (+ 1261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:09:28.140516 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:09:28.140528 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:09:28.141026 (+ 498us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:09:28.141123 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:09:28.141149 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:09:28.141271 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:28.141526 (+ 255us) server_negotiation.cc:299] Negotiation successful
0808 04:09:28.141682 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":345,"threads_started":1}
I20260808 04:09:28.144126 5041 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 73971c7c2782457497b052fd1f54be46: Neither blocks nor log segments found. Creating new log.
I20260808 04:09:28.145316 5041 log.cc:824] T 00000000000000000000000000000000 P 73971c7c2782457497b052fd1f54be46: Log is configured to *not* fsync() on all Append() calls
I20260808 04:09:28.146377 5452 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:09:28.148217 5041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73971c7c2782457497b052fd1f54be46: No bootstrap required, opened a new log
I20260808 04:09:28.157706 5044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:28.151369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45053 (local address 127.5.83.62:41953)
0808 04:09:28.151534 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:28.151540 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:28.151949 (+ 409us) server_negotiation.cc:407] Connection header received
0808 04:09:28.152094 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:28.152099 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:28.152170 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:28.152278 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:28.152669 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:28.153882 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:28.155147 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:28.155885 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:28.156015 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:28.156999 (+ 984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:09:28.157006 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:09:28.157010 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:09:28.157029 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:09:28.157072 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:09:28.157094 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:09:28.157203 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:28.157329 (+ 126us) server_negotiation.cc:299] Negotiation successful
0808 04:09:28.157377 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 04:09:28.157455 5041 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73971c7c2782457497b052fd1f54be46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73971c7c2782457497b052fd1f54be46" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 41953 } }
I20260808 04:09:28.157876 5041 raft_consensus.cc:374] T 00000000000000000000000000000000 P 73971c7c2782457497b052fd1f54be46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:09:28.157981 5041 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73971c7c2782457497b052fd1f54be46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73971c7c2782457497b052fd1f54be46, State: Initialized, Role: FOLLOWER
I20260808 04:09:28.158574 5041 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73971c7c2782457497b052fd1f54be46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73971c7c2782457497b052fd1f54be46" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 41953 } }
I20260808 04:09:28.158849 5041 raft_consensus.cc:388] T 00000000000000000000000000000000 P 73971c7c2782457497b052fd1f54be46 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:09:28.158989 5041 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73971c7c2782457497b052fd1f54be46 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:09:28.159144 5041 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73971c7c2782457497b052fd1f54be46 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:09:28.161218 5041 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73971c7c2782457497b052fd1f54be46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73971c7c2782457497b052fd1f54be46" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 41953 } }
I20260808 04:09:28.161665 5041 leader_election.cc:302] T 00000000000000000000000000000000 P 73971c7c2782457497b052fd1f54be46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73971c7c2782457497b052fd1f54be46; no voters:
I20260808 04:09:28.162174 5041 leader_election.cc:288] T 00000000000000000000000000000000 P 73971c7c2782457497b052fd1f54be46 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:09:28.162289 5053 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 73971c7c2782457497b052fd1f54be46 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:09:28.162943 5053 raft_consensus.cc:686] T 00000000000000000000000000000000 P 73971c7c2782457497b052fd1f54be46 [term 1 LEADER]: Becoming Leader. State: Replica: 73971c7c2782457497b052fd1f54be46, State: Running, Role: LEADER
I20260808 04:09:28.163564 5053 consensus_queue.cc:237] T 00000000000000000000000000000000 P 73971c7c2782457497b052fd1f54be46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73971c7c2782457497b052fd1f54be46" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 41953 } }
I20260808 04:09:28.165117 5041 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73971c7c2782457497b052fd1f54be46 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:09:28.166208 5054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73971c7c2782457497b052fd1f54be46 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73971c7c2782457497b052fd1f54be46. Latest consensus state: current_term: 1 leader_uuid: "73971c7c2782457497b052fd1f54be46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73971c7c2782457497b052fd1f54be46" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 41953 } } }
I20260808 04:09:28.166184 5055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73971c7c2782457497b052fd1f54be46 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "73971c7c2782457497b052fd1f54be46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73971c7c2782457497b052fd1f54be46" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 41953 } } }
I20260808 04:09:28.166677 5055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73971c7c2782457497b052fd1f54be46 [sys.catalog]: This master's current role is: LEADER
I20260808 04:09:28.168087 5054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73971c7c2782457497b052fd1f54be46 [sys.catalog]: This master's current role is: LEADER
I20260808 04:09:28.168227 5058 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:09:28.173624 5058 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:09:28.182900 5058 catalog_manager.cc:1347] Generated new cluster ID: 0b4fae28642746e18ae3a6819a187f13
I20260808 04:09:28.183007 5058 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:09:28.190454 5058 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:09:28.191567 5058 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:09:28.200145 5058 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 73971c7c2782457497b052fd1f54be46: Generated new TSK 0
I20260808 04:09:28.200840 5058 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:09:28.212008 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 4952
W20260808 04:09:28.213238 5035 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:57721
W20260808 04:09:28.213995 5452 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 4952: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 04:09:28.214119 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 4952
W20260808 04:09:28.234047 4933 connection.cc:535] client connection to 127.5.83.62:41953 recv error: Network error: failed to read from TLS socket (remote: 127.5.83.62:41953): Connection reset by peer (error 104)
2026-08-08T04:09:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1996 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786161826605655-5452-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:09:28 dist-test-slave-1l3l krb5kdc[5081](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:09:28 dist-test-slave-1l3l krb5kdc[5081](info): set up 2 sockets
Aug 08 04:09:28 dist-test-slave-1l3l krb5kdc[5081](info): commencing operation
krb5kdc: starting...
W20260808 04:09:29.413875 5452 mini_kdc.cc:121] Time spent starting KDC: real 1.161s user 0.007s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786161826605655-5452-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786161826605655-5452-0/krb5kdc/test-user.keytab.
Aug 08 04:09:29 dist-test-slave-1l3l krb5kdc[5081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162169, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:09:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:09:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.83.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab.
Entry for principal kudu/127.5.83.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab.
I20260808 04:09:29.676669 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786161826605655-5452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786161826605655-5452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786161826605655-5452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:34479
--webserver_interface=127.5.83.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab
--principal=kudu/127.5.83.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:44031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.62:34479
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:09:29.993119 5100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:09:29.993382 5100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:09:29.993515 5100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:09:30.000555 5100 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:09:30.000651 5100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:09:30.000723 5100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:09:30.000798 5100 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:09:30.000854 5100 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:09:30.010195 5100 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:44031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786161826605655-5452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786161826605655-5452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.62:34479
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab
--principal=kudu/127.5.83.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:34479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786161826605655-5452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:09:30.011355 5100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:09:30.012584 5100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:09:30.023550 5106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:09:30.023511 5107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:09:30.023581 5109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:09:30.024302 5100 server_base.cc:1034] running on GCE node
I20260808 04:09:30.024996 5100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:09:30.026255 5100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:09:30.027477 5100 hybrid_clock.cc:648] HybridClock initialized: now 1786162170027426 us; error 41 us; skew 500 ppm
I20260808 04:09:30.027843 5100 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:09:30 dist-test-slave-1l3l krb5kdc[5081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162170, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:09:30.033528 5100 init.cc:377] Logged in from keytab as kudu/127.5.83.62@KRBTEST.COM (short username kudu)
I20260808 04:09:30.034832 5100 webserver.cc:460] Webserver started at http://127.5.83.62:45827/ using document root <none> and password file <none>
I20260808 04:09:30.035480 5100 fs_manager.cc:359] Metadata directory not provided
I20260808 04:09:30.035604 5100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:09:30.035928 5100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:09:30.038720 5100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786161826605655-5452-0/minicluster-data/master-0/data/instance:
uuid: "7a9ac4f4f68d48da813f867725329aa6"
format_stamp: "Formatted at 2026-08-08 04:09:30 on dist-test-slave-1l3l"
server_key: "bd50d9b32de58a6acc9797a3f069daf0"
server_key_iv: "e8c7e16282edd56be1762f9b942bf9cf"
server_key_version: "encryptionkey@0"
I20260808 04:09:30.039525 5100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786161826605655-5452-0/minicluster-data/master-0/wal/instance:
uuid: "7a9ac4f4f68d48da813f867725329aa6"
format_stamp: "Formatted at 2026-08-08 04:09:30 on dist-test-slave-1l3l"
server_key: "bd50d9b32de58a6acc9797a3f069daf0"
server_key_iv: "e8c7e16282edd56be1762f9b942bf9cf"
server_key_version: "encryptionkey@0"
I20260808 04:09:30.044317 5100 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.002s
I20260808 04:09:30.047377 5117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:09:30.048605 5100 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 04:09:30.048759 5100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786161826605655-5452-0/minicluster-data/master-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786161826605655-5452-0/minicluster-data/master-0/wal
uuid: "7a9ac4f4f68d48da813f867725329aa6"
format_stamp: "Formatted at 2026-08-08 04:09:30 on dist-test-slave-1l3l"
server_key: "bd50d9b32de58a6acc9797a3f069daf0"
server_key_iv: "e8c7e16282edd56be1762f9b942bf9cf"
server_key_version: "encryptionkey@0"
I20260808 04:09:30.048924 5100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786161826605655-5452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786161826605655-5452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786161826605655-5452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:09:30.054006 5100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:09:30.057580 5100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:09:30.057835 5100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:09:30.073660 5100 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.62:34479
I20260808 04:09:30.073701 5189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.62:34479 every 8 connection(s)
I20260808 04:09:30.074927 5100 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
I20260808 04:09:30.078287 5190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:09:30.079864 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 5100
I20260808 04:09:30.080030 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786161826605655-5452-0/minicluster-data/master-0/wal/instance
I20260808 04:09:30.080446 5452 external_mini_cluster.cc:1428] Setting key 977af39907cfa040e6bdbd89da43f0da
I20260808 04:09:30.087563 5194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:30.082024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53495 (local address 127.5.83.62:34479)
0808 04:09:30.082899 (+ 875us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:30.082914 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:30.082960 (+ 46us) server_negotiation.cc:407] Connection header received
0808 04:09:30.084513 (+ 1553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:30.084534 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:30.084844 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:30.085144 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:30.086159 (+ 1015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:09:30.086165 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:09:30.086177 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:09:30.086190 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:09:30.086248 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:09:30.086266 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:09:30.086445 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:30.086538 (+ 93us) server_negotiation.cc:299] Negotiation successful
0808 04:09:30.086687 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":693,"thread_start_us":379,"threads_started":1}
I20260808 04:09:30.091159 5190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a9ac4f4f68d48da813f867725329aa6: Bootstrap starting.
I20260808 04:09:30.091285 5452 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:09:30.094726 5190 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a9ac4f4f68d48da813f867725329aa6: Neither blocks nor log segments found. Creating new log.
I20260808 04:09:30.096004 5190 log.cc:824] T 00000000000000000000000000000000 P 7a9ac4f4f68d48da813f867725329aa6: Log is configured to *not* fsync() on all Append() calls
I20260808 04:09:30.098735 5194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:30.096213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44063 (local address 127.5.83.62:34479)
0808 04:09:30.096370 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:30.096375 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:30.096860 (+ 485us) server_negotiation.cc:407] Connection header received
0808 04:09:30.097122 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:30.097126 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:30.097169 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:30.097244 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:30.098126 (+ 882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:09:30.098132 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:09:30.098137 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:09:30.098155 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:09:30.098199 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:09:30.098214 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:09:30.098307 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:30.098430 (+ 123us) server_negotiation.cc:299] Negotiation successful
0808 04:09:30.098486 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 04:09:30.098850 5190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a9ac4f4f68d48da813f867725329aa6: No bootstrap required, opened a new log
I20260808 04:09:30.106004 5190 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a9ac4f4f68d48da813f867725329aa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a9ac4f4f68d48da813f867725329aa6" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 34479 } }
I20260808 04:09:30.106324 5190 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a9ac4f4f68d48da813f867725329aa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:09:30.106405 5190 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a9ac4f4f68d48da813f867725329aa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a9ac4f4f68d48da813f867725329aa6, State: Initialized, Role: FOLLOWER
I20260808 04:09:30.106889 5190 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a9ac4f4f68d48da813f867725329aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a9ac4f4f68d48da813f867725329aa6" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 34479 } }
I20260808 04:09:30.107090 5190 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a9ac4f4f68d48da813f867725329aa6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:09:30.107184 5190 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a9ac4f4f68d48da813f867725329aa6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:09:30.107299 5190 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a9ac4f4f68d48da813f867725329aa6 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:09:30.109328 5190 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a9ac4f4f68d48da813f867725329aa6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a9ac4f4f68d48da813f867725329aa6" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 34479 } }
I20260808 04:09:30.109850 5190 leader_election.cc:302] T 00000000000000000000000000000000 P 7a9ac4f4f68d48da813f867725329aa6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a9ac4f4f68d48da813f867725329aa6; no voters:
I20260808 04:09:30.110366 5190 leader_election.cc:288] T 00000000000000000000000000000000 P 7a9ac4f4f68d48da813f867725329aa6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:09:30.110489 5201 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a9ac4f4f68d48da813f867725329aa6 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:09:30.111078 5201 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a9ac4f4f68d48da813f867725329aa6 [term 1 LEADER]: Becoming Leader. State: Replica: 7a9ac4f4f68d48da813f867725329aa6, State: Running, Role: LEADER
I20260808 04:09:30.111574 5201 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a9ac4f4f68d48da813f867725329aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a9ac4f4f68d48da813f867725329aa6" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 34479 } }
I20260808 04:09:30.112279 5190 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a9ac4f4f68d48da813f867725329aa6 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:09:30.115020 5203 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a9ac4f4f68d48da813f867725329aa6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a9ac4f4f68d48da813f867725329aa6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a9ac4f4f68d48da813f867725329aa6" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 34479 } } }
I20260808 04:09:30.115419 5203 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a9ac4f4f68d48da813f867725329aa6 [sys.catalog]: This master's current role is: LEADER
I20260808 04:09:30.114949 5204 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a9ac4f4f68d48da813f867725329aa6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a9ac4f4f68d48da813f867725329aa6. Latest consensus state: current_term: 1 leader_uuid: "7a9ac4f4f68d48da813f867725329aa6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a9ac4f4f68d48da813f867725329aa6" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 34479 } } }
I20260808 04:09:30.116127 5204 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a9ac4f4f68d48da813f867725329aa6 [sys.catalog]: This master's current role is: LEADER
I20260808 04:09:30.117038 5213 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:09:30.122095 5213 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:09:30.130396 5213 catalog_manager.cc:1347] Generated new cluster ID: 8f447ba41ecd4bc0b8426f8dcb004c46
I20260808 04:09:30.130488 5213 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:09:30.143188 5213 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:09:30.143549 5213 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:09:30.149829 5213 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a9ac4f4f68d48da813f867725329aa6: Generated new TSK 0
I20260808 04:09:30.150481 5213 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:09:30.162047 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 5100
W20260808 04:09:30.163033 5180 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:53495
W20260808 04:09:30.163626 5452 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 5100: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 04:09:30.163794 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 5100
2026-08-08T04:09:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1952 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786161826605655-5452-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:09:30 dist-test-slave-1l3l krb5kdc[5234](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:09:30 dist-test-slave-1l3l krb5kdc[5234](info): set up 2 sockets
Aug 08 04:09:30 dist-test-slave-1l3l krb5kdc[5234](info): commencing operation
krb5kdc: starting...
W20260808 04:09:31.366964 5452 mini_kdc.cc:121] Time spent starting KDC: real 1.162s user 0.003s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786161826605655-5452-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786161826605655-5452-0/krb5kdc/test-user.keytab.
Aug 08 04:09:31 dist-test-slave-1l3l krb5kdc[5234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:09:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:09:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.83.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab.
Entry for principal kudu/127.5.83.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab.
I20260808 04:09:31.640287 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786161826605655-5452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786161826605655-5452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786161826605655-5452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:40867
--webserver_interface=127.5.83.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab
--principal=kudu/127.5.83.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:35689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.62:40867
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:09:31.958781 5251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:09:31.959074 5251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:09:31.959218 5251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:09:31.966255 5251 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:09:31.966377 5251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:09:31.966468 5251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:09:31.966549 5251 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:09:31.966611 5251 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:09:31.976114 5251 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:35689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786161826605655-5452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786161826605655-5452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.62:40867
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab
--principal=kudu/127.5.83.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:40867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786161826605655-5452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:09:31.977316 5251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:09:31.978560 5251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:09:31.989610 5257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:09:31.989646 5260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:09:31.989564 5258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:09:31.990638 5251 server_base.cc:1034] running on GCE node
I20260808 04:09:31.991321 5251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:09:31.992607 5251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:09:31.993793 5251 hybrid_clock.cc:648] HybridClock initialized: now 1786162171993752 us; error 32 us; skew 500 ppm
I20260808 04:09:31.994102 5251 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:09:31 dist-test-slave-1l3l krb5kdc[5234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162171, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:09:31.997146 5251 init.cc:377] Logged in from keytab as kudu/127.5.83.62@KRBTEST.COM (short username kudu)
I20260808 04:09:31.998534 5251 webserver.cc:460] Webserver started at http://127.5.83.62:34545/ using document root <none> and password file <none>
I20260808 04:09:31.999146 5251 fs_manager.cc:359] Metadata directory not provided
I20260808 04:09:31.999258 5251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:09:31.999544 5251 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:09:32.002460 5251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786161826605655-5452-0/minicluster-data/master-0/data/instance:
uuid: "a51325ded6da4a8b9631085ed56d53f2"
format_stamp: "Formatted at 2026-08-08 04:09:31 on dist-test-slave-1l3l"
server_key: "32203bc513b6cb1bfda74eb1a7e832f9"
server_key_iv: "ff4922a721f725897a4510d82b860341"
server_key_version: "encryptionkey@0"
I20260808 04:09:32.003198 5251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786161826605655-5452-0/minicluster-data/master-0/wal/instance:
uuid: "a51325ded6da4a8b9631085ed56d53f2"
format_stamp: "Formatted at 2026-08-08 04:09:31 on dist-test-slave-1l3l"
server_key: "32203bc513b6cb1bfda74eb1a7e832f9"
server_key_iv: "ff4922a721f725897a4510d82b860341"
server_key_version: "encryptionkey@0"
I20260808 04:09:32.008198 5251 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260808 04:09:32.011437 5269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:09:32.012701 5251 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:09:32.012868 5251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786161826605655-5452-0/minicluster-data/master-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786161826605655-5452-0/minicluster-data/master-0/wal
uuid: "a51325ded6da4a8b9631085ed56d53f2"
format_stamp: "Formatted at 2026-08-08 04:09:31 on dist-test-slave-1l3l"
server_key: "32203bc513b6cb1bfda74eb1a7e832f9"
server_key_iv: "ff4922a721f725897a4510d82b860341"
server_key_version: "encryptionkey@0"
I20260808 04:09:32.013041 5251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786161826605655-5452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786161826605655-5452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786161826605655-5452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:09:32.024276 5251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:09:32.027878 5251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:09:32.028122 5251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:09:32.044660 5251 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.62:40867
I20260808 04:09:32.044699 5339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.62:40867 every 8 connection(s)
I20260808 04:09:32.045948 5251 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
I20260808 04:09:32.046476 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 5251
I20260808 04:09:32.046643 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786161826605655-5452-0/minicluster-data/master-0/wal/instance
I20260808 04:09:32.047186 5452 external_mini_cluster.cc:1428] Setting key 180a11ef399ce131d78d649b8dc218d3
I20260808 04:09:32.049983 5340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 08 04:09:32 dist-test-slave-1l3l krb5kdc[5234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:09:32.062351 5340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a51325ded6da4a8b9631085ed56d53f2: Bootstrap starting.
I20260808 04:09:32.065341 5340 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a51325ded6da4a8b9631085ed56d53f2: Neither blocks nor log segments found. Creating new log.
I20260808 04:09:32.066303 5340 log.cc:824] T 00000000000000000000000000000000 P a51325ded6da4a8b9631085ed56d53f2: Log is configured to *not* fsync() on all Append() calls
I20260808 04:09:32.068905 5346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:32.048866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43765 (local address 127.5.83.62:40867)
0808 04:09:32.049608 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:32.049618 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:32.049656 (+ 38us) server_negotiation.cc:407] Connection header received
0808 04:09:32.051212 (+ 1556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:32.051233 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:32.051568 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:32.051980 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:32.053722 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:32.055078 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:32.056854 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:32.057508 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:32.057636 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:32.062311 (+ 4675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:09:32.062318 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:09:32.062330 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:09:32.062372 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:09:32.065173 (+ 2801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:09:32.065848 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:09:32.065854 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:09:32.065860 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:09:32.066021 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:09:32.066388 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:09:32.066391 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:09:32.066394 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:09:32.066912 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:09:32.067212 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:32.067606 (+ 394us) server_negotiation.cc:299] Negotiation successful
0808 04:09:32.068015 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":330,"threads_started":1}
I20260808 04:09:32.069135 5340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a51325ded6da4a8b9631085ed56d53f2: No bootstrap required, opened a new log
I20260808 04:09:32.072487 5452 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:09:32.077135 5340 raft_consensus.cc:348] T 00000000000000000000000000000000 P a51325ded6da4a8b9631085ed56d53f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a51325ded6da4a8b9631085ed56d53f2" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 40867 } }
I20260808 04:09:32.077481 5340 raft_consensus.cc:374] T 00000000000000000000000000000000 P a51325ded6da4a8b9631085ed56d53f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:09:32.077576 5340 raft_consensus.cc:729] T 00000000000000000000000000000000 P a51325ded6da4a8b9631085ed56d53f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a51325ded6da4a8b9631085ed56d53f2, State: Initialized, Role: FOLLOWER
I20260808 04:09:32.078126 5340 consensus_queue.cc:260] T 00000000000000000000000000000000 P a51325ded6da4a8b9631085ed56d53f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a51325ded6da4a8b9631085ed56d53f2" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 40867 } }
I20260808 04:09:32.078341 5340 raft_consensus.cc:388] T 00000000000000000000000000000000 P a51325ded6da4a8b9631085ed56d53f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:09:32.078471 5340 raft_consensus.cc:482] T 00000000000000000000000000000000 P a51325ded6da4a8b9631085ed56d53f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:09:32.078615 5340 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a51325ded6da4a8b9631085ed56d53f2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:09:32.080794 5340 raft_consensus.cc:504] T 00000000000000000000000000000000 P a51325ded6da4a8b9631085ed56d53f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a51325ded6da4a8b9631085ed56d53f2" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 40867 } }
I20260808 04:09:32.081267 5340 leader_election.cc:302] T 00000000000000000000000000000000 P a51325ded6da4a8b9631085ed56d53f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a51325ded6da4a8b9631085ed56d53f2; no voters:
I20260808 04:09:32.081844 5340 leader_election.cc:288] T 00000000000000000000000000000000 P a51325ded6da4a8b9631085ed56d53f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:09:32.081941 5357 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a51325ded6da4a8b9631085ed56d53f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:09:32.083926 5357 raft_consensus.cc:686] T 00000000000000000000000000000000 P a51325ded6da4a8b9631085ed56d53f2 [term 1 LEADER]: Becoming Leader. State: Replica: a51325ded6da4a8b9631085ed56d53f2, State: Running, Role: LEADER
I20260808 04:09:32.084182 5340 sys_catalog.cc:564] T 00000000000000000000000000000000 P a51325ded6da4a8b9631085ed56d53f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:09:32.084551 5357 consensus_queue.cc:237] T 00000000000000000000000000000000 P a51325ded6da4a8b9631085ed56d53f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a51325ded6da4a8b9631085ed56d53f2" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 40867 } }
I20260808 04:09:32.089437 5346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:32.076452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44573 (local address 127.5.83.62:40867)
0808 04:09:32.076574 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:32.076578 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:32.077150 (+ 572us) server_negotiation.cc:407] Connection header received
0808 04:09:32.077289 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:32.077293 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:32.077350 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:32.077428 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:32.078806 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:32.079549 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:32.080788 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:32.081239 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:32.081351 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:32.083783 (+ 2432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:09:32.083789 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:09:32.083797 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:09:32.083833 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:09:32.086879 (+ 3046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:09:32.087506 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:09:32.087512 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:09:32.087515 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:09:32.087634 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:09:32.088030 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:09:32.088035 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:09:32.088039 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:09:32.088337 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:09:32.088598 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:32.088878 (+ 280us) server_negotiation.cc:299] Negotiation successful
0808 04:09:32.089074 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 04:09:32.093216 5358 sys_catalog.cc:455] T 00000000000000000000000000000000 P a51325ded6da4a8b9631085ed56d53f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a51325ded6da4a8b9631085ed56d53f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a51325ded6da4a8b9631085ed56d53f2" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 40867 } } }
I20260808 04:09:32.093169 5359 sys_catalog.cc:455] T 00000000000000000000000000000000 P a51325ded6da4a8b9631085ed56d53f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a51325ded6da4a8b9631085ed56d53f2. Latest consensus state: current_term: 1 leader_uuid: "a51325ded6da4a8b9631085ed56d53f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a51325ded6da4a8b9631085ed56d53f2" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 40867 } } }
I20260808 04:09:32.093678 5358 sys_catalog.cc:458] T 00000000000000000000000000000000 P a51325ded6da4a8b9631085ed56d53f2 [sys.catalog]: This master's current role is: LEADER
I20260808 04:09:32.093705 5359 sys_catalog.cc:458] T 00000000000000000000000000000000 P a51325ded6da4a8b9631085ed56d53f2 [sys.catalog]: This master's current role is: LEADER
I20260808 04:09:32.096729 5366 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:09:32.101511 5366 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:09:32.110036 5366 catalog_manager.cc:1347] Generated new cluster ID: 13b51fdd5be14cfcb2b68ddf1507f517
I20260808 04:09:32.110139 5366 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:09:32.123369 5366 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:09:32.124586 5366 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:09:32.139328 5366 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a51325ded6da4a8b9631085ed56d53f2: Generated new TSK 0
I20260808 04:09:32.140062 5366 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:09:32.156546 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 5251
I20260808 04:09:32.157862 5331 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:09:32.246554 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 5251
2026-08-08T04:09:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2080 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786161826605655-5452-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:09:32 dist-test-slave-1l3l krb5kdc[5386](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:09:32 dist-test-slave-1l3l krb5kdc[5386](info): set up 2 sockets
Aug 08 04:09:32 dist-test-slave-1l3l krb5kdc[5386](info): commencing operation
krb5kdc: starting...
W20260808 04:09:33.438330 5452 mini_kdc.cc:121] Time spent starting KDC: real 1.152s user 0.000s sys 0.037s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786161826605655-5452-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786161826605655-5452-0/krb5kdc/test-user.keytab.
Aug 08 04:09:33 dist-test-slave-1l3l krb5kdc[5386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162173, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:09:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:09:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.83.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab.
Entry for principal kudu/127.5.83.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab.
I20260808 04:09:33.707161 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786161826605655-5452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786161826605655-5452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786161826605655-5452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:38059
--webserver_interface=127.5.83.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab
--principal=kudu/127.5.83.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:39461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.62:38059
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:09:34.022437 5403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:09:34.022720 5403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:09:34.022869 5403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:09:34.029760 5403 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:09:34.029878 5403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:09:34.029955 5403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:09:34.030036 5403 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:09:34.030095 5403 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:09:34.039251 5403 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:39461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786161826605655-5452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786161826605655-5452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.62:38059
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab
--principal=kudu/127.5.83.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:38059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786161826605655-5452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:09:34.040436 5403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:09:34.041630 5403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:09:34.052194 5408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:09:34.052327 5411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:09:34.052332 5409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:09:34.053352 5403 server_base.cc:1034] running on GCE node
I20260808 04:09:34.054075 5403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:09:34.055276 5403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:09:34.056478 5403 hybrid_clock.cc:648] HybridClock initialized: now 1786162174056428 us; error 41 us; skew 500 ppm
I20260808 04:09:34.056854 5403 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:09:34 dist-test-slave-1l3l krb5kdc[5386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162174, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:09:34.059911 5403 init.cc:377] Logged in from keytab as kudu/127.5.83.62@KRBTEST.COM (short username kudu)
I20260808 04:09:34.061197 5403 webserver.cc:460] Webserver started at http://127.5.83.62:33895/ using document root <none> and password file <none>
I20260808 04:09:34.061805 5403 fs_manager.cc:359] Metadata directory not provided
I20260808 04:09:34.061905 5403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:09:34.062143 5403 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:09:34.064913 5403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786161826605655-5452-0/minicluster-data/master-0/data/instance:
uuid: "2301f5502e0244758fdcd693281ff69f"
format_stamp: "Formatted at 2026-08-08 04:09:34 on dist-test-slave-1l3l"
server_key: "226b65eba34d2e872fbcdb1d165dc4aa"
server_key_iv: "64da184c86a3909a33ae31561be2a841"
server_key_version: "encryptionkey@0"
I20260808 04:09:34.065603 5403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786161826605655-5452-0/minicluster-data/master-0/wal/instance:
uuid: "2301f5502e0244758fdcd693281ff69f"
format_stamp: "Formatted at 2026-08-08 04:09:34 on dist-test-slave-1l3l"
server_key: "226b65eba34d2e872fbcdb1d165dc4aa"
server_key_iv: "64da184c86a3909a33ae31561be2a841"
server_key_version: "encryptionkey@0"
I20260808 04:09:34.070374 5403 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 04:09:34.073537 5424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:09:34.074703 5403 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 04:09:34.074834 5403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786161826605655-5452-0/minicluster-data/master-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786161826605655-5452-0/minicluster-data/master-0/wal
uuid: "2301f5502e0244758fdcd693281ff69f"
format_stamp: "Formatted at 2026-08-08 04:09:34 on dist-test-slave-1l3l"
server_key: "226b65eba34d2e872fbcdb1d165dc4aa"
server_key_iv: "64da184c86a3909a33ae31561be2a841"
server_key_version: "encryptionkey@0"
I20260808 04:09:34.074993 5403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786161826605655-5452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786161826605655-5452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786161826605655-5452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:09:34.092691 5403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:09:34.096573 5403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:09:34.096786 5403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:09:34.112989 5403 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.62:38059
I20260808 04:09:34.113034 5501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.62:38059 every 8 connection(s)
I20260808 04:09:34.114185 5403 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
I20260808 04:09:34.117623 5502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:09:34.120020 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 5403
I20260808 04:09:34.120172 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786161826605655-5452-0/minicluster-data/master-0/wal/instance
I20260808 04:09:34.120627 5452 external_mini_cluster.cc:1428] Setting key 08414fc1896704ad0596f1373c77ee80
I20260808 04:09:34.129936 5502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2301f5502e0244758fdcd693281ff69f: Bootstrap starting.
I20260808 04:09:34.133534 5502 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2301f5502e0244758fdcd693281ff69f: Neither blocks nor log segments found. Creating new log.
I20260808 04:09:34.134807 5502 log.cc:824] T 00000000000000000000000000000000 P 2301f5502e0244758fdcd693281ff69f: Log is configured to *not* fsync() on all Append() calls
I20260808 04:09:34.135066 5506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:34.122360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52453 (local address 127.5.83.62:38059)
0808 04:09:34.123111 (+ 751us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:34.123125 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:34.123174 (+ 49us) server_negotiation.cc:407] Connection header received
0808 04:09:34.125430 (+ 2256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:34.125456 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:34.125933 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:34.126345 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:34.126839 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:34.128845 (+ 2006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:34.130231 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:34.131191 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:34.131353 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:34.132453 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:09:34.132464 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:09:34.132481 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:09:34.133133 (+ 652us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:09:34.133229 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:09:34.133261 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:09:34.133378 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:34.133544 (+ 166us) server_negotiation.cc:299] Negotiation successful
0808 04:09:34.133788 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":292,"threads_started":1}
I20260808 04:09:34.137641 5502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2301f5502e0244758fdcd693281ff69f: No bootstrap required, opened a new log
I20260808 04:09:34.138602 5452 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:09:34.145498 5502 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2301f5502e0244758fdcd693281ff69f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2301f5502e0244758fdcd693281ff69f" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 38059 } }
I20260808 04:09:34.145803 5502 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2301f5502e0244758fdcd693281ff69f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:09:34.145910 5502 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2301f5502e0244758fdcd693281ff69f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2301f5502e0244758fdcd693281ff69f, State: Initialized, Role: FOLLOWER
I20260808 04:09:34.146494 5502 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2301f5502e0244758fdcd693281ff69f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2301f5502e0244758fdcd693281ff69f" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 38059 } }
I20260808 04:09:34.146782 5502 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2301f5502e0244758fdcd693281ff69f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:09:34.146934 5502 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2301f5502e0244758fdcd693281ff69f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:09:34.147122 5502 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2301f5502e0244758fdcd693281ff69f [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:09:34.148694 5506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:34.142722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48629 (local address 127.5.83.62:38059)
0808 04:09:34.142891 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:34.142897 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:34.143221 (+ 324us) server_negotiation.cc:407] Connection header received
0808 04:09:34.143339 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:34.143343 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:34.143427 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:34.143529 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:34.143996 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:34.145055 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:34.146308 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:34.146950 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:34.147079 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:34.148019 (+ 940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:09:34.148025 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:09:34.148030 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:09:34.148049 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:09:34.148097 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:09:34.148116 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:09:34.148211 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:34.148289 (+ 78us) server_negotiation.cc:299] Negotiation successful
0808 04:09:34.148374 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 04:09:34.149712 5502 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2301f5502e0244758fdcd693281ff69f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2301f5502e0244758fdcd693281ff69f" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 38059 } }
I20260808 04:09:34.150164 5502 leader_election.cc:302] T 00000000000000000000000000000000 P 2301f5502e0244758fdcd693281ff69f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2301f5502e0244758fdcd693281ff69f; no voters:
I20260808 04:09:34.150686 5502 leader_election.cc:288] T 00000000000000000000000000000000 P 2301f5502e0244758fdcd693281ff69f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:09:34.150791 5513 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2301f5502e0244758fdcd693281ff69f [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:09:34.151378 5513 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2301f5502e0244758fdcd693281ff69f [term 1 LEADER]: Becoming Leader. State: Replica: 2301f5502e0244758fdcd693281ff69f, State: Running, Role: LEADER
I20260808 04:09:34.151893 5513 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2301f5502e0244758fdcd693281ff69f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2301f5502e0244758fdcd693281ff69f" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 38059 } }
I20260808 04:09:34.152552 5502 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2301f5502e0244758fdcd693281ff69f [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:09:34.154175 5515 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2301f5502e0244758fdcd693281ff69f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2301f5502e0244758fdcd693281ff69f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2301f5502e0244758fdcd693281ff69f" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 38059 } } }
I20260808 04:09:34.154465 5515 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2301f5502e0244758fdcd693281ff69f [sys.catalog]: This master's current role is: LEADER
I20260808 04:09:34.155220 5521 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:09:34.155530 5516 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2301f5502e0244758fdcd693281ff69f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2301f5502e0244758fdcd693281ff69f. Latest consensus state: current_term: 1 leader_uuid: "2301f5502e0244758fdcd693281ff69f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2301f5502e0244758fdcd693281ff69f" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 38059 } } }
I20260808 04:09:34.155921 5516 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2301f5502e0244758fdcd693281ff69f [sys.catalog]: This master's current role is: LEADER
I20260808 04:09:34.160000 5521 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:09:34.170516 5521 catalog_manager.cc:1347] Generated new cluster ID: 937039925c3e462089c3e5ab2cf59e5a
I20260808 04:09:34.170608 5521 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:09:34.185501 5521 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:09:34.186693 5521 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:09:34.196427 5521 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2301f5502e0244758fdcd693281ff69f: Generated new TSK 0
I20260808 04:09:34.197196 5521 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:09:34.211807 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 5403
W20260808 04:09:34.212818 5495 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:52453
W20260808 04:09:34.213518 5452 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 5403: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 04:09:34.213647 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 5403
2026-08-08T04:09:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1966 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786161826605655-5452-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:09:34 dist-test-slave-1l3l krb5kdc[5544](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:09:34 dist-test-slave-1l3l krb5kdc[5544](info): set up 2 sockets
Aug 08 04:09:34 dist-test-slave-1l3l krb5kdc[5544](info): commencing operation
krb5kdc: starting...
W20260808 04:09:35.425197 5452 mini_kdc.cc:121] Time spent starting KDC: real 1.172s user 0.000s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786161826605655-5452-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786161826605655-5452-0/krb5kdc/test-user.keytab.
Aug 08 04:09:35 dist-test-slave-1l3l krb5kdc[5544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:09:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:09:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.83.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab.
Entry for principal kudu/127.5.83.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab.
I20260808 04:09:35.698187 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786161826605655-5452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786161826605655-5452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786161826605655-5452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:46179
--webserver_interface=127.5.83.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab
--principal=kudu/127.5.83.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:42189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.62:46179
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:09:36.012871 5567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:09:36.013134 5567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:09:36.013269 5567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:09:36.020179 5567 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:09:36.020282 5567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:09:36.020363 5567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:09:36.020449 5567 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:09:36.020509 5567 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:09:36.029915 5567 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:42189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786161826605655-5452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786161826605655-5452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.62:46179
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab
--principal=kudu/127.5.83.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:46179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786161826605655-5452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:09:36.031091 5567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:09:36.032353 5567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:09:36.043162 5575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:09:36.043222 5574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:09:36.043231 5577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:09:36.043977 5567 server_base.cc:1034] running on GCE node
I20260808 04:09:36.044698 5567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:09:36.045951 5567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:09:36.047154 5567 hybrid_clock.cc:648] HybridClock initialized: now 1786162176047113 us; error 37 us; skew 500 ppm
I20260808 04:09:36.047472 5567 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:09:36 dist-test-slave-1l3l krb5kdc[5544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162176, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:09:36.055405 5567 init.cc:377] Logged in from keytab as kudu/127.5.83.62@KRBTEST.COM (short username kudu)
I20260808 04:09:36.056866 5567 webserver.cc:460] Webserver started at http://127.5.83.62:46383/ using document root <none> and password file <none>
I20260808 04:09:36.057523 5567 fs_manager.cc:359] Metadata directory not provided
I20260808 04:09:36.057639 5567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:09:36.057925 5567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:09:36.060812 5567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786161826605655-5452-0/minicluster-data/master-0/data/instance:
uuid: "327ff421aec94e03a58331055d8bd0f8"
format_stamp: "Formatted at 2026-08-08 04:09:36 on dist-test-slave-1l3l"
server_key: "916099d74ddae93ce3abb8270a334fd1"
server_key_iv: "ddaf4390577ffe75b2808cbb00b66a3e"
server_key_version: "encryptionkey@0"
I20260808 04:09:36.061601 5567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786161826605655-5452-0/minicluster-data/master-0/wal/instance:
uuid: "327ff421aec94e03a58331055d8bd0f8"
format_stamp: "Formatted at 2026-08-08 04:09:36 on dist-test-slave-1l3l"
server_key: "916099d74ddae93ce3abb8270a334fd1"
server_key_iv: "ddaf4390577ffe75b2808cbb00b66a3e"
server_key_version: "encryptionkey@0"
I20260808 04:09:36.066399 5567 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 04:09:36.069633 5584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:09:36.070996 5567 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 04:09:36.071151 5567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786161826605655-5452-0/minicluster-data/master-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786161826605655-5452-0/minicluster-data/master-0/wal
uuid: "327ff421aec94e03a58331055d8bd0f8"
format_stamp: "Formatted at 2026-08-08 04:09:36 on dist-test-slave-1l3l"
server_key: "916099d74ddae93ce3abb8270a334fd1"
server_key_iv: "ddaf4390577ffe75b2808cbb00b66a3e"
server_key_version: "encryptionkey@0"
I20260808 04:09:36.071317 5567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786161826605655-5452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786161826605655-5452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786161826605655-5452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:09:36.076663 5567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:09:36.080425 5567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:09:36.080677 5567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:09:36.096251 5567 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.62:46179
I20260808 04:09:36.096285 5660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.62:46179 every 8 connection(s)
I20260808 04:09:36.097545 5567 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
I20260808 04:09:36.100924 5661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:09:36.101006 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 5567
I20260808 04:09:36.101169 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786161826605655-5452-0/minicluster-data/master-0/wal/instance
I20260808 04:09:36.101588 5452 external_mini_cluster.cc:1428] Setting key bb4ab3fd67f0c316c981920d201965fb
I20260808 04:09:36.109110 5665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:36.103141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53981 (local address 127.5.83.62:46179)
0808 04:09:36.103975 (+ 834us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:36.103988 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:36.104041 (+ 53us) server_negotiation.cc:407] Connection header received
0808 04:09:36.105535 (+ 1494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:36.105562 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:36.105978 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:36.106377 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:36.107292 (+ 915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:09:36.107303 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:09:36.107318 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:09:36.107339 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:09:36.107406 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:09:36.107435 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:09:36.107657 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:36.107809 (+ 152us) server_negotiation.cc:299] Negotiation successful
0808 04:09:36.107997 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":335,"threads_started":1}
I20260808 04:09:36.112586 5452 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:09:36.114492 5661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 327ff421aec94e03a58331055d8bd0f8: Bootstrap starting.
I20260808 04:09:36.117587 5661 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 327ff421aec94e03a58331055d8bd0f8: Neither blocks nor log segments found. Creating new log.
I20260808 04:09:36.118697 5665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:36.116515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43737 (local address 127.5.83.62:46179)
0808 04:09:36.116663 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:36.116668 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:36.117126 (+ 458us) server_negotiation.cc:407] Connection header received
0808 04:09:36.117226 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:36.117230 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:36.117281 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:36.117379 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:36.118149 (+ 770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:09:36.118154 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:09:36.118158 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:09:36.118174 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:09:36.118215 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:09:36.118233 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:09:36.118344 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:36.118394 (+ 50us) server_negotiation.cc:299] Negotiation successful
0808 04:09:36.118442 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 04:09:36.118701 5661 log.cc:824] T 00000000000000000000000000000000 P 327ff421aec94e03a58331055d8bd0f8: Log is configured to *not* fsync() on all Append() calls
I20260808 04:09:36.121238 5661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 327ff421aec94e03a58331055d8bd0f8: No bootstrap required, opened a new log
I20260808 04:09:36.128161 5661 raft_consensus.cc:348] T 00000000000000000000000000000000 P 327ff421aec94e03a58331055d8bd0f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "327ff421aec94e03a58331055d8bd0f8" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 46179 } }
I20260808 04:09:36.128480 5661 raft_consensus.cc:374] T 00000000000000000000000000000000 P 327ff421aec94e03a58331055d8bd0f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:09:36.128554 5661 raft_consensus.cc:729] T 00000000000000000000000000000000 P 327ff421aec94e03a58331055d8bd0f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 327ff421aec94e03a58331055d8bd0f8, State: Initialized, Role: FOLLOWER
I20260808 04:09:36.129024 5661 consensus_queue.cc:260] T 00000000000000000000000000000000 P 327ff421aec94e03a58331055d8bd0f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "327ff421aec94e03a58331055d8bd0f8" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 46179 } }
I20260808 04:09:36.129225 5661 raft_consensus.cc:388] T 00000000000000000000000000000000 P 327ff421aec94e03a58331055d8bd0f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:09:36.129315 5661 raft_consensus.cc:482] T 00000000000000000000000000000000 P 327ff421aec94e03a58331055d8bd0f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:09:36.129438 5661 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 327ff421aec94e03a58331055d8bd0f8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:09:36.131423 5661 raft_consensus.cc:504] T 00000000000000000000000000000000 P 327ff421aec94e03a58331055d8bd0f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "327ff421aec94e03a58331055d8bd0f8" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 46179 } }
I20260808 04:09:36.131999 5661 leader_election.cc:302] T 00000000000000000000000000000000 P 327ff421aec94e03a58331055d8bd0f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 327ff421aec94e03a58331055d8bd0f8; no voters:
I20260808 04:09:36.132565 5661 leader_election.cc:288] T 00000000000000000000000000000000 P 327ff421aec94e03a58331055d8bd0f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:09:36.132675 5674 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 327ff421aec94e03a58331055d8bd0f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:09:36.133265 5674 raft_consensus.cc:686] T 00000000000000000000000000000000 P 327ff421aec94e03a58331055d8bd0f8 [term 1 LEADER]: Becoming Leader. State: Replica: 327ff421aec94e03a58331055d8bd0f8, State: Running, Role: LEADER
I20260808 04:09:36.133762 5674 consensus_queue.cc:237] T 00000000000000000000000000000000 P 327ff421aec94e03a58331055d8bd0f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "327ff421aec94e03a58331055d8bd0f8" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 46179 } }
I20260808 04:09:36.134321 5661 sys_catalog.cc:564] T 00000000000000000000000000000000 P 327ff421aec94e03a58331055d8bd0f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:09:36.136446 5675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 327ff421aec94e03a58331055d8bd0f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "327ff421aec94e03a58331055d8bd0f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "327ff421aec94e03a58331055d8bd0f8" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 46179 } } }
I20260808 04:09:36.136968 5675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 327ff421aec94e03a58331055d8bd0f8 [sys.catalog]: This master's current role is: LEADER
I20260808 04:09:36.137262 5676 sys_catalog.cc:455] T 00000000000000000000000000000000 P 327ff421aec94e03a58331055d8bd0f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 327ff421aec94e03a58331055d8bd0f8. Latest consensus state: current_term: 1 leader_uuid: "327ff421aec94e03a58331055d8bd0f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "327ff421aec94e03a58331055d8bd0f8" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 46179 } } }
I20260808 04:09:36.137553 5676 sys_catalog.cc:458] T 00000000000000000000000000000000 P 327ff421aec94e03a58331055d8bd0f8 [sys.catalog]: This master's current role is: LEADER
I20260808 04:09:36.140836 5685 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:09:36.144738 5685 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:09:36.152743 5685 catalog_manager.cc:1347] Generated new cluster ID: 86abd20e0539496f881889554562fc7a
I20260808 04:09:36.152848 5685 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:09:36.168022 5685 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:09:36.168362 5685 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:09:36.174501 5685 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 327ff421aec94e03a58331055d8bd0f8: Generated new TSK 0
I20260808 04:09:36.175205 5685 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:09:36.179759 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 5567
W20260808 04:09:36.180992 5654 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:53981
W20260808 04:09:36.181609 5452 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 5567: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 04:09:36.181767 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 5567
2026-08-08T04:09:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1966 ms)
[----------] 18 tests from AuthTokenIssuingTest (36314 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161826605655-5452-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:09:36 dist-test-slave-1l3l krb5kdc[5701](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:09:36 dist-test-slave-1l3l krb5kdc[5701](info): set up 2 sockets
Aug 08 04:09:36 dist-test-slave-1l3l krb5kdc[5701](info): commencing operation
krb5kdc: starting...
W20260808 04:09:37.383106 5452 mini_kdc.cc:121] Time spent starting KDC: real 1.163s user 0.001s sys 0.039s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161826605655-5452-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161826605655-5452-0/krb5kdc/test-user.keytab.
Aug 08 04:09:37 dist-test-slave-1l3l krb5kdc[5701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:09:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:09:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.83.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab.
Entry for principal kudu/127.5.83.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab.
I20260808 04:09:37.673389 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161826605655-5452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161826605655-5452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161826605655-5452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:45311
--webserver_interface=127.5.83.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab
--principal=kudu/127.5.83.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:42369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.62:45311,127.5.83.61:32879,127.5.83.60:36219
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:09:37.991952 5718 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 04:09:37.992270 5718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:09:37.992367 5718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:09:37.992491 5718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:09:37.999377 5718 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:09:37.999470 5718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:09:37.999552 5718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:09:37.999614 5718 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:09:37.999698 5718 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:09:38.009202 5718 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:42369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161826605655-5452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161826605655-5452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.62:45311,127.5.83.61:32879,127.5.83.60:36219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab
--principal=kudu/127.5.83.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:45311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161826605655-5452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:09:38.010414 5718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:09:38.011719 5718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:09:38.022696 5725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:09:38.022760 5726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:09:38.023211 5729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:09:38.023440 5718 server_base.cc:1034] running on GCE node
I20260808 04:09:38.024216 5718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:09:38.025472 5718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:09:38.026671 5718 hybrid_clock.cc:648] HybridClock initialized: now 1786162178026610 us; error 38 us; skew 500 ppm
I20260808 04:09:38.026938 5718 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:09:38 dist-test-slave-1l3l krb5kdc[5701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162178, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:09:38.029810 5718 init.cc:377] Logged in from keytab as kudu/127.5.83.62@KRBTEST.COM (short username kudu)
I20260808 04:09:38.031082 5718 webserver.cc:460] Webserver started at http://127.5.83.62:37593/ using document root <none> and password file <none>
I20260808 04:09:38.031688 5718 fs_manager.cc:359] Metadata directory not provided
I20260808 04:09:38.031824 5718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:09:38.032135 5718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:09:38.034878 5718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161826605655-5452-0/minicluster-data/master-0/data/instance:
uuid: "adc7645fa8bc478b9e11d8df76225fdf"
format_stamp: "Formatted at 2026-08-08 04:09:38 on dist-test-slave-1l3l"
server_key: "6ba84f6c1bd5c324d8b593f8138f9f90"
server_key_iv: "2da1d8fa73055fb92791940d84103222"
server_key_version: "encryptionkey@0"
I20260808 04:09:38.035633 5718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161826605655-5452-0/minicluster-data/master-0/wal/instance:
uuid: "adc7645fa8bc478b9e11d8df76225fdf"
format_stamp: "Formatted at 2026-08-08 04:09:38 on dist-test-slave-1l3l"
server_key: "6ba84f6c1bd5c324d8b593f8138f9f90"
server_key_iv: "2da1d8fa73055fb92791940d84103222"
server_key_version: "encryptionkey@0"
I20260808 04:09:38.040426 5718 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260808 04:09:38.043476 5737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:09:38.044682 5718 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 04:09:38.044838 5718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161826605655-5452-0/minicluster-data/master-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161826605655-5452-0/minicluster-data/master-0/wal
uuid: "adc7645fa8bc478b9e11d8df76225fdf"
format_stamp: "Formatted at 2026-08-08 04:09:38 on dist-test-slave-1l3l"
server_key: "6ba84f6c1bd5c324d8b593f8138f9f90"
server_key_iv: "2da1d8fa73055fb92791940d84103222"
server_key_version: "encryptionkey@0"
I20260808 04:09:38.045030 5718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161826605655-5452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161826605655-5452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161826605655-5452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:09:38.098958 5718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:09:38.102964 5718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:09:38.103214 5718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:09:38.120428 5718 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.62:45311
I20260808 04:09:38.120450 5814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.62:45311 every 8 connection(s)
I20260808 04:09:38.121649 5718 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
I20260808 04:09:38.125331 5815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:09:38.129982 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 5718
I20260808 04:09:38.130175 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161826605655-5452-0/minicluster-data/master-0/wal/instance
I20260808 04:09:38.130800 5452 external_mini_cluster.cc:1428] Setting key 4182654631ffe90ef29fb9d239a5b5ba
I20260808 04:09:38.131642 5815 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.83.62" port: 45311 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 04:09:38 dist-test-slave-1l3l krb5kdc[5701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162178, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.62@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:09:38.153326 5819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:38.135943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39132 (local address 127.5.83.62:45311)
0808 04:09:38.136624 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:38.136636 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:38.136686 (+ 50us) server_negotiation.cc:407] Connection header received
0808 04:09:38.137812 (+ 1126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:38.137817 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:38.138014 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:38.138192 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:38.139419 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:38.140641 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:38.141962 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:38.142487 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:38.142653 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:38.146522 (+ 3869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:09:38.146530 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:09:38.146541 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:09:38.146569 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:09:38.149395 (+ 2826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:09:38.150082 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:09:38.150090 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:09:38.150093 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:09:38.150231 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:09:38.150586 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:09:38.150591 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:09:38.150595 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:09:38.151029 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:09:38.151216 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:38.151733 (+ 517us) server_negotiation.cc:299] Negotiation successful
0808 04:09:38.151990 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":491,"thread_start_us":296,"threads_started":1}
I20260808 04:09:38.153343 5817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:38.134725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:45311 (local address 127.0.0.1:39132)
0808 04:09:38.135471 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0808 04:09:38.135516 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 04:09:38.137475 (+ 1959us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:09:38.138235 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:09:38.138241 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:09:38.138906 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:09:38.139215 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:09:38.139231 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:09:38.140780 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:09:38.140784 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:09:38.141832 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:09:38.141844 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:09:38.142706 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:09:38.142713 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:09:38.142955 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:38.143569 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:09:38.143593 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:09:38.146243 (+ 2650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:09:38.149497 (+ 3254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:09:38.149507 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:09:38.149518 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:09:38.149977 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:09:38.150332 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:09:38.150337 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:09:38.150341 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:09:38.150490 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:09:38.151159 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:09:38.151166 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:09:38.151356 (+ 190us) client_negotiation.cc:769] Sending connection context
0808 04:09:38.151568 (+ 212us) client_negotiation.cc:240] Negotiation successful
0808 04:09:38.152119 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"mutex_wait_us":182,"thread_start_us":311,"threads_started":1}
I20260808 04:09:38.156427 5815 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.83.61" port: 32879 } has no permanent_uuid. Determining permanent_uuid...
I20260808 04:09:38.157501 5817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:38.157054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.61:32879 (local address 127.0.0.1:33618)
0808 04:09:38.157230 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0808 04:09:38.157357 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.61:32879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260808 04:09:38.157806 5742 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.83.61:32879: connect: Connection refused (error 111)
I20260808 04:09:38.159356 5817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:38.158947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.61:32879 (local address 127.0.0.1:33630)
0808 04:09:38.159114 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0808 04:09:38.159198 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.61:32879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260808 04:09:38.159550 5815 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.83.61:32879: Network error: Client connection negotiation failed: client connection to 127.5.83.61:32879: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.5.83.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.61.keytab.
Entry for principal kudu/127.5.83.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.61.keytab.
I20260808 04:09:38.194332 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161826605655-5452-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161826605655-5452-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161826605655-5452-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161826605655-5452-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.61:32879
--webserver_interface=127.5.83.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.61.keytab
--principal=kudu/127.5.83.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:42369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.62:45311,127.5.83.61:32879,127.5.83.60:36219
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:09:38.208861 5815 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.83.61" port: 32879 } attempt: 1
I20260808 04:09:38.210034 5817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:38.209597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.61:32879 (local address 127.0.0.1:33636)
0808 04:09:38.209779 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0808 04:09:38.209866 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.61:32879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260808 04:09:38.210997 5817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:38.210660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.61:32879 (local address 127.0.0.1:33646)
0808 04:09:38.210804 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0808 04:09:38.210868 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.61:32879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260808 04:09:38.211208 5815 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.83.61:32879: Network error: Client connection negotiation failed: client connection to 127.5.83.61:32879: connect: Connection refused (error 111)
I20260808 04:09:38.279454 5815 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.83.61" port: 32879 } attempt: 2
I20260808 04:09:38.280718 5817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:38.280107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.61:32879 (local address 127.0.0.1:49480)
0808 04:09:38.280404 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0808 04:09:38.280531 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.61:32879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260808 04:09:38.281824 5817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:38.281393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.61:32879 (local address 127.0.0.1:49490)
0808 04:09:38.281556 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0808 04:09:38.281661 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.61:32879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260808 04:09:38.282029 5815 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.83.61:32879: Network error: Client connection negotiation failed: client connection to 127.5.83.61:32879: connect: Connection refused (error 111)
I20260808 04:09:38.373324 5815 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.83.61" port: 32879 } attempt: 3
I20260808 04:09:38.374619 5817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:38.374057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.61:32879 (local address 127.0.0.1:49498)
0808 04:09:38.374307 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0808 04:09:38.374434 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.61:32879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260808 04:09:38.375686 5817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:38.375268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.61:32879 (local address 127.0.0.1:49500)
0808 04:09:38.375419 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0808 04:09:38.375520 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.61:32879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260808 04:09:38.375939 5815 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.83.61:32879: Network error: Client connection negotiation failed: client connection to 127.5.83.61:32879: connect: Connection refused (error 111)
W20260808 04:09:38.517531 5823 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 04:09:38.517830 5823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:09:38.517903 5823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:09:38.517983 5823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260808 04:09:38.519198 5815 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.83.61" port: 32879 } attempt: 4
I20260808 04:09:38.520442 5817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:38.519849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.61:32879 (local address 127.0.0.1:49508)
0808 04:09:38.520173 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0808 04:09:38.520280 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.61:32879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160}
I20260808 04:09:38.521461 5817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:38.521072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.61:32879 (local address 127.0.0.1:49524)
0808 04:09:38.521215 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0808 04:09:38.521313 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.61:32879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260808 04:09:38.521656 5815 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.83.61:32879: Network error: Client connection negotiation failed: client connection to 127.5.83.61:32879: connect: Connection refused (error 111)
W20260808 04:09:38.525746 5823 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:09:38.525823 5823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:09:38.525890 5823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:09:38.525942 5823 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:09:38.526005 5823 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:09:38.535521 5823 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:42369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161826605655-5452-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161826605655-5452-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.62:45311,127.5.83.61:32879,127.5.83.60:36219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.61.keytab
--principal=kudu/127.5.83.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.61:32879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161826605655-5452-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.5.83.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161826605655-5452-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:09:38.536826 5823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:09:38.538034 5823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:09:38.549320 5829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:09:38.549472 5832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:09:38.549367 5830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:09:38.550172 5823 server_base.cc:1034] running on GCE node
I20260808 04:09:38.550942 5823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:09:38.552270 5823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:09:38.553498 5823 hybrid_clock.cc:648] HybridClock initialized: now 1786162178553430 us; error 28 us; skew 500 ppm
I20260808 04:09:38.553781 5823 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:09:38 dist-test-slave-1l3l krb5kdc[5701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162178, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:09:38.556730 5823 init.cc:377] Logged in from keytab as kudu/127.5.83.61@KRBTEST.COM (short username kudu)
I20260808 04:09:38.558010 5823 webserver.cc:460] Webserver started at http://127.5.83.61:41061/ using document root <none> and password file <none>
I20260808 04:09:38.558630 5823 fs_manager.cc:359] Metadata directory not provided
I20260808 04:09:38.558730 5823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:09:38.559028 5823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:09:38.561918 5823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161826605655-5452-0/minicluster-data/master-1/data/instance:
uuid: "68ebee93c4e844e19dc106d167ca7ee8"
format_stamp: "Formatted at 2026-08-08 04:09:38 on dist-test-slave-1l3l"
server_key: "68f657f13549cbfd9d862827e4f99024"
server_key_iv: "cf39299c82d0095422ea26fa448d3363"
server_key_version: "encryptionkey@0"
I20260808 04:09:38.562659 5823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161826605655-5452-0/minicluster-data/master-1/wal/instance:
uuid: "68ebee93c4e844e19dc106d167ca7ee8"
format_stamp: "Formatted at 2026-08-08 04:09:38 on dist-test-slave-1l3l"
server_key: "68f657f13549cbfd9d862827e4f99024"
server_key_iv: "cf39299c82d0095422ea26fa448d3363"
server_key_version: "encryptionkey@0"
I20260808 04:09:38.567376 5823 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260808 04:09:38.570518 5841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:09:38.571660 5823 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:09:38.571836 5823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161826605655-5452-0/minicluster-data/master-1/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161826605655-5452-0/minicluster-data/master-1/wal
uuid: "68ebee93c4e844e19dc106d167ca7ee8"
format_stamp: "Formatted at 2026-08-08 04:09:38 on dist-test-slave-1l3l"
server_key: "68f657f13549cbfd9d862827e4f99024"
server_key_iv: "cf39299c82d0095422ea26fa448d3363"
server_key_version: "encryptionkey@0"
I20260808 04:09:38.572043 5823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161826605655-5452-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161826605655-5452-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161826605655-5452-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:09:38.585947 5823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:09:38.589494 5823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:09:38.589730 5823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:09:38.606998 5823 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.61:32879
I20260808 04:09:38.607019 5909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.61:32879 every 8 connection(s)
I20260808 04:09:38.608337 5823 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161826605655-5452-0/minicluster-data/master-1/data/info.pb
I20260808 04:09:38.610311 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 5823
I20260808 04:09:38.610476 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161826605655-5452-0/minicluster-data/master-1/wal/instance
I20260808 04:09:38.610905 5452 external_mini_cluster.cc:1428] Setting key 42dc7ddb1f63e1d7b7ac020dced3ba0e
I20260808 04:09:38.612000 5910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:09:38.618146 5910 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.83.62" port: 45311 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 04:09:38 dist-test-slave-1l3l krb5kdc[5701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162178, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.61@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:09:38.638463 5819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:38.620609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51568 (local address 127.5.83.62:45311)
0808 04:09:38.620801 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:38.620806 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:38.621943 (+ 1137us) server_negotiation.cc:407] Connection header received
0808 04:09:38.623856 (+ 1913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:38.623861 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:38.623930 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:38.624020 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:38.625812 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:38.626494 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:38.628548 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:38.628963 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:38.629057 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:38.632948 (+ 3891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:09:38.632953 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:09:38.632956 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:09:38.632989 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:09:38.635480 (+ 2491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:09:38.636356 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:09:38.636360 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:09:38.636362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:09:38.636499 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:09:38.636897 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:09:38.636899 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:09:38.636901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:09:38.637157 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:09:38.637274 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:38.638045 (+ 771us) server_negotiation.cc:299] Negotiation successful
0808 04:09:38.638217 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260808 04:09:38.639830 5912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:38.621032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:45311 (local address 127.0.0.1:51568)
0808 04:09:38.621769 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0808 04:09:38.621812 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 04:09:38.623544 (+ 1732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:09:38.624255 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:09:38.624265 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:09:38.624951 (+ 686us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:09:38.625657 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:09:38.625680 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:09:38.626649 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:09:38.626654 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:09:38.628410 (+ 1756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:09:38.628426 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:09:38.629178 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:09:38.629185 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:09:38.629448 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:38.630128 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:09:38.630154 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:09:38.632727 (+ 2573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:09:38.635611 (+ 2884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:09:38.635619 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:09:38.635633 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:09:38.636261 (+ 628us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:09:38.636591 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:09:38.636596 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:09:38.636599 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:09:38.636809 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:09:38.637286 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:09:38.637294 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:09:38.637687 (+ 393us) client_negotiation.cc:769] Sending connection context
0808 04:09:38.637971 (+ 284us) client_negotiation.cc:240] Negotiation successful
0808 04:09:38.638454 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":320,"threads_started":1}
I20260808 04:09:38.641631 5910 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.83.61" port: 32879 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.5.83.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.60@KRBTEST.COM" created.
Aug 08 04:09:38 dist-test-slave-1l3l krb5kdc[5701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162178, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.61@KRBTEST.COM for kudu/127.5.83.61@KRBTEST.COM
I20260808 04:09:38.656045 5912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:38.642501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.61:32879 (local address 127.0.0.1:49532)
0808 04:09:38.642689 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0808 04:09:38.642709 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 04:09:38.642823 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:09:38.643594 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:09:38.643597 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:09:38.644067 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:09:38.644218 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:09:38.644227 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:09:38.645613 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:09:38.645619 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:09:38.646791 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:09:38.646808 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:09:38.647649 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:09:38.647659 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:09:38.647859 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:38.648404 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:09:38.648424 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:09:38.650629 (+ 2205us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:09:38.653677 (+ 3048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:09:38.653681 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:09:38.653685 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:09:38.654047 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:09:38.654394 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:09:38.654399 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:09:38.654402 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:09:38.654519 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:09:38.655114 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:09:38.655118 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:09:38.655256 (+ 138us) client_negotiation.cc:769] Sending connection context
0808 04:09:38.655393 (+ 137us) client_negotiation.cc:240] Negotiation successful
0808 04:09:38.655654 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260808 04:09:38.656335 5915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:38.642522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49532 (local address 127.5.83.61:32879)
0808 04:09:38.643154 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:38.643161 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:38.643193 (+ 32us) server_negotiation.cc:407] Connection header received
0808 04:09:38.643244 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:38.643253 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:38.643450 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:38.643564 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:38.644366 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:38.645397 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:38.646949 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:38.647474 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:38.647577 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:38.650823 (+ 3246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:09:38.650833 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:09:38.650844 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:09:38.650882 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:09:38.653560 (+ 2678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:09:38.654170 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:09:38.654178 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:09:38.654180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:09:38.654288 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:09:38.654660 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:09:38.654663 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:09:38.654665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:09:38.654992 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:09:38.655176 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:38.655548 (+ 372us) server_negotiation.cc:299] Negotiation successful
0808 04:09:38.655812 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":295,"threads_started":1}
I20260808 04:09:38.658598 5910 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.83.60" port: 36219 } has no permanent_uuid. Determining permanent_uuid...
I20260808 04:09:38.659842 5912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:38.659318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.60:36219 (local address 127.0.0.1:33872)
0808 04:09:38.659508 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0808 04:09:38.659631 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.60:36219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260808 04:09:38.660147 5845 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.83.60:36219: connect: Connection refused (error 111)
I20260808 04:09:38.661700 5912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:38.661339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.60:36219 (local address 127.0.0.1:33876)
0808 04:09:38.661487 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0808 04:09:38.661585 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.60:36219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260808 04:09:38.661924 5910 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.83.60:36219: Network error: Client connection negotiation failed: client connection to 127.5.83.60:36219: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.60.keytab.
Entry for principal kudu/127.5.83.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.60.keytab.
I20260808 04:09:38.680989 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161826605655-5452-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161826605655-5452-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161826605655-5452-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161826605655-5452-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.60:36219
--webserver_interface=127.5.83.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.60.keytab
--principal=kudu/127.5.83.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:42369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.62:45311,127.5.83.61:32879,127.5.83.60:36219
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:09:38.711201 5910 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.83.60" port: 36219 } attempt: 1
I20260808 04:09:38.712379 5912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:38.711918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.60:36219 (local address 127.0.0.1:33884)
0808 04:09:38.712139 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0808 04:09:38.712228 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.60:36219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260808 04:09:38.713316 5912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:38.712981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.60:36219 (local address 127.0.0.1:33896)
0808 04:09:38.713124 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0808 04:09:38.713211 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.60:36219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260808 04:09:38.713533 5910 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.83.60:36219: Network error: Client connection negotiation failed: client connection to 127.5.83.60:36219: connect: Connection refused (error 111)
I20260808 04:09:38.781737 5910 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.83.60" port: 36219 } attempt: 2
I20260808 04:09:38.782760 5912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:38.782325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.60:36219 (local address 127.0.0.1:33900)
0808 04:09:38.782516 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0808 04:09:38.782612 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.60:36219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260808 04:09:38.783751 5912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:38.783385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.60:36219 (local address 127.0.0.1:33916)
0808 04:09:38.783535 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0808 04:09:38.783599 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.60:36219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260808 04:09:38.783955 5910 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.83.60:36219: Network error: Client connection negotiation failed: client connection to 127.5.83.60:36219: connect: Connection refused (error 111)
I20260808 04:09:38.820921 5815 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.83.61" port: 32879 } attempt: 5
Aug 08 04:09:38 dist-test-slave-1l3l krb5kdc[5701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162178, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.62@KRBTEST.COM for kudu/127.5.83.61@KRBTEST.COM
I20260808 04:09:38.834173 5915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:38.821789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49548 (local address 127.5.83.61:32879)
0808 04:09:38.821974 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:38.821980 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:38.822002 (+ 22us) server_negotiation.cc:407] Connection header received
0808 04:09:38.822133 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:38.822138 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:38.822219 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:38.822307 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:38.823141 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:38.823892 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:38.825041 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:38.825496 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:38.825605 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:38.829105 (+ 3500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:09:38.829110 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:09:38.829114 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:09:38.829146 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:09:38.831614 (+ 2468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:09:38.832428 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:09:38.832432 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:09:38.832434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:09:38.832560 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:09:38.832957 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:09:38.832960 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:09:38.832962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:09:38.833228 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:09:38.833336 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:38.833767 (+ 431us) server_negotiation.cc:299] Negotiation successful
0808 04:09:38.833944 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 04:09:38.834371 5817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:38.821645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.61:32879 (local address 127.0.0.1:49548)
0808 04:09:38.821823 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0808 04:09:38.821845 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 04:09:38.822004 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:09:38.822398 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:09:38.822404 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:09:38.822806 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:09:38.822993 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:09:38.823002 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:09:38.824045 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:09:38.824049 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:09:38.824890 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:09:38.824912 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:09:38.825633 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:09:38.825638 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:09:38.825833 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:38.826404 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:09:38.826423 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:09:38.828917 (+ 2494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:09:38.831779 (+ 2862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:09:38.831784 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:09:38.831788 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:09:38.832306 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:09:38.832664 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:09:38.832669 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:09:38.832673 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:09:38.832852 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:09:38.833338 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:09:38.833343 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:09:38.833533 (+ 190us) client_negotiation.cc:769] Sending connection context
0808 04:09:38.833696 (+ 163us) client_negotiation.cc:240] Negotiation successful
0808 04:09:38.834009 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260808 04:09:38.835573 5815 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.83.60" port: 36219 } has no permanent_uuid. Determining permanent_uuid...
I20260808 04:09:38.836764 5817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:38.836326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.60:36219 (local address 127.0.0.1:33930)
0808 04:09:38.836505 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0808 04:09:38.836595 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.60:36219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260808 04:09:38.837740 5817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:38.837376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.60:36219 (local address 127.0.0.1:33936)
0808 04:09:38.837512 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0808 04:09:38.837617 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.60:36219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260808 04:09:38.837951 5815 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.83.60:36219: Network error: Client connection negotiation failed: client connection to 127.5.83.60:36219: connect: Connection refused (error 111)
I20260808 04:09:38.875149 5910 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.83.60" port: 36219 } attempt: 3
I20260808 04:09:38.876363 5912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:38.875851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.60:36219 (local address 127.0.0.1:33938)
0808 04:09:38.876065 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0808 04:09:38.876189 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.60:36219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260808 04:09:38.877403 5912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:38.876994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.60:36219 (local address 127.0.0.1:33954)
0808 04:09:38.877203 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0808 04:09:38.877283 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.60:36219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
W20260808 04:09:38.877607 5910 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.83.60:36219: Network error: Client connection negotiation failed: client connection to 127.5.83.60:36219: connect: Connection refused (error 111)
I20260808 04:09:38.889153 5815 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.83.60" port: 36219 } attempt: 1
I20260808 04:09:38.890177 5817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:38.889663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.60:36219 (local address 127.0.0.1:33962)
0808 04:09:38.889873 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0808 04:09:38.889990 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.60:36219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260808 04:09:38.891171 5817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:38.890796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.60:36219 (local address 127.0.0.1:33966)
0808 04:09:38.890989 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0808 04:09:38.891058 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.60:36219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260808 04:09:38.891378 5815 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.83.60:36219: Network error: Client connection negotiation failed: client connection to 127.5.83.60:36219: connect: Connection refused (error 111)
I20260808 04:09:38.959565 5815 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.83.60" port: 36219 } attempt: 2
I20260808 04:09:38.960611 5817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:38.960163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.60:36219 (local address 127.0.0.1:33968)
0808 04:09:38.960367 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0808 04:09:38.960452 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.60:36219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260808 04:09:38.961608 5817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:38.961305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.60:36219 (local address 127.0.0.1:33976)
0808 04:09:38.961435 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0808 04:09:38.961505 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.60:36219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260808 04:09:38.961812 5815 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.83.60:36219: Network error: Client connection negotiation failed: client connection to 127.5.83.60:36219: connect: Connection refused (error 111)
W20260808 04:09:39.006484 5919 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 04:09:39.006785 5919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:09:39.006858 5919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:09:39.006938 5919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:09:39.013983 5919 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:09:39.014060 5919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:09:39.014114 5919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:09:39.014163 5919 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:09:39.014222 5919 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:09:39.020785 5910 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.83.60" port: 36219 } attempt: 4
I20260808 04:09:39.021835 5912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:39.021359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.60:36219 (local address 127.0.0.1:33992)
0808 04:09:39.021591 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0808 04:09:39.021681 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.60:36219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260808 04:09:39.022871 5912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:39.022548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.60:36219 (local address 127.0.0.1:34000)
0808 04:09:39.022704 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0808 04:09:39.022770 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.60:36219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260808 04:09:39.023087 5910 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.83.60:36219: Network error: Client connection negotiation failed: client connection to 127.5.83.60:36219: connect: Connection refused (error 111)
I20260808 04:09:39.023775 5919 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:42369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161826605655-5452-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161826605655-5452-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.62:45311,127.5.83.61:32879,127.5.83.60:36219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.60.keytab
--principal=kudu/127.5.83.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.60:36219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161826605655-5452-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.5.83.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161826605655-5452-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:09:39.025152 5919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:09:39.026762 5919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:09:39.038278 5926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:09:39.038233 5929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:09:39.038398 5925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:09:39.039212 5919 server_base.cc:1034] running on GCE node
I20260808 04:09:39.039983 5919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:09:39.041208 5919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:09:39.042423 5919 hybrid_clock.cc:648] HybridClock initialized: now 1786162179042358 us; error 37 us; skew 500 ppm
I20260808 04:09:39.042711 5919 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:09:39 dist-test-slave-1l3l krb5kdc[5701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162179, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:09:39.045662 5919 init.cc:377] Logged in from keytab as kudu/127.5.83.60@KRBTEST.COM (short username kudu)
I20260808 04:09:39.047008 5919 webserver.cc:460] Webserver started at http://127.5.83.60:33711/ using document root <none> and password file <none>
I20260808 04:09:39.047626 5919 fs_manager.cc:359] Metadata directory not provided
I20260808 04:09:39.047753 5919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:09:39.048094 5919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:09:39.050949 5919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161826605655-5452-0/minicluster-data/master-2/data/instance:
uuid: "d0897e4e6b774803b14a1609c99b2594"
format_stamp: "Formatted at 2026-08-08 04:09:39 on dist-test-slave-1l3l"
server_key: "7eeb855820ae8ec696a41b4054e67622"
server_key_iv: "b6888c68f78fec5ab08021215402d8f9"
server_key_version: "encryptionkey@0"
I20260808 04:09:39.051651 5919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161826605655-5452-0/minicluster-data/master-2/wal/instance:
uuid: "d0897e4e6b774803b14a1609c99b2594"
format_stamp: "Formatted at 2026-08-08 04:09:39 on dist-test-slave-1l3l"
server_key: "7eeb855820ae8ec696a41b4054e67622"
server_key_iv: "b6888c68f78fec5ab08021215402d8f9"
server_key_version: "encryptionkey@0"
I20260808 04:09:39.056365 5919 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 04:09:39.059448 5937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:09:39.060686 5919 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260808 04:09:39.060834 5919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161826605655-5452-0/minicluster-data/master-2/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161826605655-5452-0/minicluster-data/master-2/wal
uuid: "d0897e4e6b774803b14a1609c99b2594"
format_stamp: "Formatted at 2026-08-08 04:09:39 on dist-test-slave-1l3l"
server_key: "7eeb855820ae8ec696a41b4054e67622"
server_key_iv: "b6888c68f78fec5ab08021215402d8f9"
server_key_version: "encryptionkey@0"
I20260808 04:09:39.061031 5919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161826605655-5452-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161826605655-5452-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161826605655-5452-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:09:39.067987 5815 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.83.60" port: 36219 } attempt: 3
I20260808 04:09:39.068938 5817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:39.068532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.60:36219 (local address 127.0.0.1:34014)
0808 04:09:39.068710 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0808 04:09:39.068791 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.60:36219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260808 04:09:39.069856 5817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:39.069500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.60:36219 (local address 127.0.0.1:34018)
0808 04:09:39.069685 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0808 04:09:39.069751 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.60:36219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260808 04:09:39.070128 5815 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.83.60:36219: Network error: Client connection negotiation failed: client connection to 127.5.83.60:36219: connect: Connection refused (error 111)
I20260808 04:09:39.083317 5919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:09:39.086798 5919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:09:39.087024 5919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:09:39.103740 5919 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.60:36219
I20260808 04:09:39.103755 6001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.60:36219 every 8 connection(s)
I20260808 04:09:39.104954 5919 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161826605655-5452-0/minicluster-data/master-2/data/info.pb
I20260808 04:09:39.106581 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 5919
I20260808 04:09:39.106772 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161826605655-5452-0/minicluster-data/master-2/wal/instance
I20260808 04:09:39.107307 5452 external_mini_cluster.cc:1428] Setting key 54c1af720a84a4ecbc8e316a7ecc5c08
I20260808 04:09:39.109244 6003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:09:39.117216 6003 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.83.62" port: 45311 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 04:09:39 dist-test-slave-1l3l krb5kdc[5701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.60@KRBTEST.COM
Aug 08 04:09:39 dist-test-slave-1l3l krb5kdc[5701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162179, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.60@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:09:39.130496 6008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:39.109097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34028 (local address 127.5.83.60:36219)
0808 04:09:39.109846 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:39.109857 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:39.109902 (+ 45us) server_negotiation.cc:407] Connection header received
0808 04:09:39.111434 (+ 1532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:39.111454 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:39.111830 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:39.112145 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:39.113893 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:39.115165 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:39.116934 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:39.117565 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:39.117698 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:39.122564 (+ 4866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:09:39.122574 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:09:39.122586 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:09:39.122632 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:09:39.125987 (+ 3355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:09:39.126746 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:09:39.126754 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:09:39.126762 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:09:39.126931 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:09:39.127337 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:09:39.127342 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:09:39.127345 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:09:39.128058 (+ 713us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:09:39.128339 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:39.129230 (+ 891us) server_negotiation.cc:299] Negotiation successful
0808 04:09:39.129547 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":381,"threads_started":1}
I20260808 04:09:39.135291 6009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:39.119801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:45311 (local address 127.0.0.1:51580)
0808 04:09:39.120698 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0808 04:09:39.120743 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 04:09:39.120946 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:09:39.121298 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:09:39.121305 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:09:39.121915 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:09:39.122139 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:09:39.122148 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:09:39.123446 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:09:39.123450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:09:39.125038 (+ 1588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:09:39.125053 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:09:39.126052 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:09:39.126059 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:09:39.126236 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:39.127145 (+ 909us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:09:39.127167 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:09:39.129355 (+ 2188us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:09:39.132630 (+ 3275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:09:39.132640 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:09:39.132647 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:09:39.133174 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:09:39.133648 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:09:39.133653 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:09:39.133656 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:09:39.133770 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:09:39.134281 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:09:39.134288 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:09:39.134466 (+ 178us) client_negotiation.cc:769] Sending connection context
0808 04:09:39.134603 (+ 137us) client_negotiation.cc:240] Negotiation successful
0808 04:09:39.134872 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":688,"thread_start_us":555,"threads_started":1}
I20260808 04:09:39.135288 5819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:39.119854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51580 (local address 127.5.83.62:45311)
0808 04:09:39.120143 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:39.120149 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:39.120823 (+ 674us) server_negotiation.cc:407] Connection header received
0808 04:09:39.121077 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:39.121081 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:39.121140 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:39.121221 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:39.122282 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:39.123282 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:39.125186 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:39.125863 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:39.125983 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:39.129623 (+ 3640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:09:39.129627 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:09:39.129630 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:09:39.129661 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:09:39.132516 (+ 2855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:09:39.133309 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:09:39.133315 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:09:39.133318 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:09:39.133499 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:09:39.133887 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:09:39.133890 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:09:39.133892 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:09:39.134156 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:09:39.134320 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:39.134754 (+ 434us) server_negotiation.cc:299] Negotiation successful
0808 04:09:39.134946 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20260808 04:09:39.136539 6003 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.83.61" port: 32879 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 04:09:39 dist-test-slave-1l3l krb5kdc[5701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
Aug 08 04:09:39 dist-test-slave-1l3l krb5kdc[5701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162179, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.60@KRBTEST.COM for kudu/127.5.83.61@KRBTEST.COM
I20260808 04:09:39.147782 6010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:39.133332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51596 (local address 127.5.83.62:45311)
0808 04:09:39.133880 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:39.133884 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:39.133910 (+ 26us) server_negotiation.cc:407] Connection header received
0808 04:09:39.133949 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:39.133952 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:39.134000 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:39.134088 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:39.135265 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:39.136304 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:39.137602 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:39.138085 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:39.138171 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:39.142563 (+ 4392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:09:39.142568 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:09:39.142571 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:09:39.142599 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:09:39.145132 (+ 2533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:09:39.145888 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:09:39.145892 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:09:39.145894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:09:39.146024 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:09:39.146422 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:09:39.146426 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:09:39.146428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:09:39.146679 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:09:39.146943 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:39.147213 (+ 270us) server_negotiation.cc:299] Negotiation successful
0808 04:09:39.147413 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":283,"threads_started":1}
I20260808 04:09:39.149082 6009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:39.137179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.61:32879 (local address 127.0.0.1:49560)
0808 04:09:39.137343 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0808 04:09:39.137360 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:09:39.137452 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:09:39.137742 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:09:39.137746 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:09:39.138046 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:09:39.138189 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:09:39.138199 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:09:39.139100 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:09:39.139105 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:09:39.140456 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:09:39.140472 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:09:39.141115 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:09:39.141120 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:09:39.141242 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:39.141717 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:09:39.141738 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:09:39.143747 (+ 2009us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:09:39.146752 (+ 3005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:09:39.146757 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:09:39.146761 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:09:39.147094 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:09:39.147428 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:09:39.147433 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:09:39.147437 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:09:39.147587 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:09:39.148219 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:09:39.148224 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:09:39.148379 (+ 155us) client_negotiation.cc:769] Sending connection context
0808 04:09:39.148533 (+ 154us) client_negotiation.cc:240] Negotiation successful
0808 04:09:39.148777 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260808 04:09:39.149195 5915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:39.137296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49560 (local address 127.5.83.61:32879)
0808 04:09:39.137457 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:39.137461 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:39.137476 (+ 15us) server_negotiation.cc:407] Connection header received
0808 04:09:39.137551 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:39.137555 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:39.137628 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:39.137724 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:39.138329 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:39.138955 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:39.140581 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:39.140976 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:39.141070 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:39.143915 (+ 2845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:09:39.143919 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:09:39.143922 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:09:39.143951 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:09:39.146642 (+ 2691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:09:39.147200 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:09:39.147204 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:09:39.147206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:09:39.147324 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:09:39.147736 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:09:39.147742 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:09:39.147744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:09:39.148055 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:09:39.148201 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:39.148653 (+ 452us) server_negotiation.cc:299] Negotiation successful
0808 04:09:39.148846 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 04:09:39.150053 6003 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.83.60" port: 36219 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 04:09:39 dist-test-slave-1l3l krb5kdc[5701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162179, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.60@KRBTEST.COM for kudu/127.5.83.60@KRBTEST.COM
Aug 08 04:09:39 dist-test-slave-1l3l krb5kdc[5701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.61@KRBTEST.COM
I20260808 04:09:39.163432 6009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:39.150689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.60:36219 (local address 127.0.0.1:34042)
0808 04:09:39.150955 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0808 04:09:39.150971 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:09:39.151077 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:09:39.151339 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:09:39.151344 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:09:39.151830 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:09:39.151963 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:09:39.151971 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:09:39.153136 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:09:39.153141 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:09:39.154479 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:09:39.154491 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:09:39.155342 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:09:39.155348 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:09:39.155511 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:39.155955 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:09:39.155978 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:09:39.158180 (+ 2202us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:09:39.160994 (+ 2814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:09:39.160999 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:09:39.161003 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:09:39.161486 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:09:39.161828 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:09:39.161833 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:09:39.161837 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:09:39.161991 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:09:39.162575 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:09:39.162578 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:09:39.162710 (+ 132us) client_negotiation.cc:769] Sending connection context
0808 04:09:39.162864 (+ 154us) client_negotiation.cc:240] Negotiation successful
0808 04:09:39.163113 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183}
I20260808 04:09:39.163664 6008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:39.150689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34042 (local address 127.5.83.60:36219)
0808 04:09:39.150851 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:39.150857 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:39.151038 (+ 181us) server_negotiation.cc:407] Connection header received
0808 04:09:39.151168 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:39.151171 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:39.151221 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:39.151289 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:39.152088 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:39.153023 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:39.154597 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:39.155178 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:39.155261 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:39.158340 (+ 3079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:09:39.158345 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:09:39.158348 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:09:39.158375 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:09:39.160868 (+ 2493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:09:39.161595 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:09:39.161603 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:09:39.161606 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:09:39.161717 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:09:39.162101 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:09:39.162105 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:09:39.162108 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:09:39.162467 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:09:39.162610 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:39.162968 (+ 358us) server_negotiation.cc:299] Negotiation successful
0808 04:09:39.163309 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":88,"server-negotiator.queue_time_us":49}
I20260808 04:09:39.164837 5915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:39.149758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49564 (local address 127.5.83.61:32879)
0808 04:09:39.150112 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:39.150117 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:39.150132 (+ 15us) server_negotiation.cc:407] Connection header received
0808 04:09:39.150172 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:39.150177 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:39.150232 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:39.150308 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:39.151794 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:39.152759 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:39.154334 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:39.154837 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:39.154964 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:39.159303 (+ 4339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:09:39.159307 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:09:39.159310 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:09:39.159337 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:09:39.162008 (+ 2671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:09:39.162705 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:09:39.162710 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:09:39.162713 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:09:39.162861 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:09:39.163286 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:09:39.163291 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:09:39.163295 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:09:39.163797 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:09:39.164033 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:39.164346 (+ 313us) server_negotiation.cc:299] Negotiation successful
0808 04:09:39.164559 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249}
I20260808 04:09:39.168654 6003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0897e4e6b774803b14a1609c99b2594: Bootstrap starting.
I20260808 04:09:39.169100 5452 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:09:39.171728 6003 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0897e4e6b774803b14a1609c99b2594: Neither blocks nor log segments found. Creating new log.
I20260808 04:09:39.172976 6003 log.cc:824] T 00000000000000000000000000000000 P d0897e4e6b774803b14a1609c99b2594: Log is configured to *not* fsync() on all Append() calls
I20260808 04:09:39.171674 5780 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "adc7645fa8bc478b9e11d8df76225fdf"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51596
I20260808 04:09:39.175973 6003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0897e4e6b774803b14a1609c99b2594: No bootstrap required, opened a new log
I20260808 04:09:39.186442 6003 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0897e4e6b774803b14a1609c99b2594 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc7645fa8bc478b9e11d8df76225fdf" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 45311 } } peers { permanent_uuid: "68ebee93c4e844e19dc106d167ca7ee8" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 32879 } } peers { permanent_uuid: "d0897e4e6b774803b14a1609c99b2594" member_type: VOTER last_known_addr { host: "127.5.83.60" port: 36219 } }
I20260808 04:09:39.186836 6003 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0897e4e6b774803b14a1609c99b2594 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0897e4e6b774803b14a1609c99b2594, State: Initialized, Role: FOLLOWER
I20260808 04:09:39.187315 6003 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0897e4e6b774803b14a1609c99b2594 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc7645fa8bc478b9e11d8df76225fdf" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 45311 } } peers { permanent_uuid: "68ebee93c4e844e19dc106d167ca7ee8" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 32879 } } peers { permanent_uuid: "d0897e4e6b774803b14a1609c99b2594" member_type: VOTER last_known_addr { host: "127.5.83.60" port: 36219 } }
I20260808 04:09:39.188215 6012 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0897e4e6b774803b14a1609c99b2594 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc7645fa8bc478b9e11d8df76225fdf" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 45311 } } peers { permanent_uuid: "68ebee93c4e844e19dc106d167ca7ee8" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 32879 } } peers { permanent_uuid: "d0897e4e6b774803b14a1609c99b2594" member_type: VOTER last_known_addr { host: "127.5.83.60" port: 36219 } } }
I20260808 04:09:39.188530 6012 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0897e4e6b774803b14a1609c99b2594 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:09:39.189424 6003 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0897e4e6b774803b14a1609c99b2594 [sys.catalog]: configured and running, proceeding with master startup.
W20260808 04:09:39.198212 6027 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d0897e4e6b774803b14a1609c99b2594: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 04:09:39.198339 6027 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 04:09:39.196368 5919 master_runner.cc:186] Error getting master registration for 127.5.83.62:45311: OK, instance_id { permanent_uuid: "adc7645fa8bc478b9e11d8df76225fdf" instance_seqno: 1786162178115529 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 04:09:39.203864 5919 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 04:09:39.247339 5815 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.83.60" port: 36219 } attempt: 4
Aug 08 04:09:39 dist-test-slave-1l3l krb5kdc[5701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162178, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.62@KRBTEST.COM for kudu/127.5.83.60@KRBTEST.COM
I20260808 04:09:39.261647 6008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:39.248337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34050 (local address 127.5.83.60:36219)
0808 04:09:39.248538 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:39.248543 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:39.248566 (+ 23us) server_negotiation.cc:407] Connection header received
0808 04:09:39.248733 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:39.248739 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:39.248821 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:39.248913 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:39.249743 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:39.250490 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:39.251738 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:39.252189 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:39.252298 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:39.255630 (+ 3332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:09:39.255635 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:09:39.255638 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:09:39.255669 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:09:39.259143 (+ 3474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:09:39.259750 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:09:39.259755 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:09:39.259759 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:09:39.259879 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:09:39.260245 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:09:39.260250 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:09:39.260254 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:09:39.260576 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:09:39.260712 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:39.261099 (+ 387us) server_negotiation.cc:299] Negotiation successful
0808 04:09:39.261292 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260808 04:09:39.261641 5817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:39.248176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.60:36219 (local address 127.0.0.1:34050)
0808 04:09:39.248387 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0808 04:09:39.248408 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 04:09:39.248574 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:09:39.248967 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:09:39.248972 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:09:39.249381 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:09:39.249595 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:09:39.249608 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:09:39.250627 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:09:39.250633 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:09:39.251571 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:09:39.251584 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:09:39.252328 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:09:39.252333 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:09:39.252520 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:39.253159 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:09:39.253175 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:09:39.255453 (+ 2278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:09:39.259259 (+ 3806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:09:39.259265 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:09:39.259269 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:09:39.259619 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:09:39.259982 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:09:39.259987 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:09:39.259991 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:09:39.260143 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:09:39.260701 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:09:39.260706 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:09:39.260854 (+ 148us) client_negotiation.cc:769] Sending connection context
0808 04:09:39.260979 (+ 125us) client_negotiation.cc:240] Negotiation successful
0808 04:09:39.261254 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260808 04:09:39.267614 5815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adc7645fa8bc478b9e11d8df76225fdf: Bootstrap starting.
I20260808 04:09:39.271492 5815 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P adc7645fa8bc478b9e11d8df76225fdf: Neither blocks nor log segments found. Creating new log.
I20260808 04:09:39.272742 5815 log.cc:824] T 00000000000000000000000000000000 P adc7645fa8bc478b9e11d8df76225fdf: Log is configured to *not* fsync() on all Append() calls
I20260808 04:09:39.275758 5815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adc7645fa8bc478b9e11d8df76225fdf: No bootstrap required, opened a new log
I20260808 04:09:39.282747 5815 raft_consensus.cc:348] T 00000000000000000000000000000000 P adc7645fa8bc478b9e11d8df76225fdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc7645fa8bc478b9e11d8df76225fdf" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 45311 } } peers { permanent_uuid: "68ebee93c4e844e19dc106d167ca7ee8" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 32879 } } peers { permanent_uuid: "d0897e4e6b774803b14a1609c99b2594" member_type: VOTER last_known_addr { host: "127.5.83.60" port: 36219 } }
I20260808 04:09:39.283121 5815 raft_consensus.cc:729] T 00000000000000000000000000000000 P adc7645fa8bc478b9e11d8df76225fdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adc7645fa8bc478b9e11d8df76225fdf, State: Initialized, Role: FOLLOWER
I20260808 04:09:39.283577 5815 consensus_queue.cc:260] T 00000000000000000000000000000000 P adc7645fa8bc478b9e11d8df76225fdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc7645fa8bc478b9e11d8df76225fdf" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 45311 } } peers { permanent_uuid: "68ebee93c4e844e19dc106d167ca7ee8" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 32879 } } peers { permanent_uuid: "d0897e4e6b774803b14a1609c99b2594" member_type: VOTER last_known_addr { host: "127.5.83.60" port: 36219 } }
I20260808 04:09:39.284516 6032 sys_catalog.cc:455] T 00000000000000000000000000000000 P adc7645fa8bc478b9e11d8df76225fdf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc7645fa8bc478b9e11d8df76225fdf" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 45311 } } peers { permanent_uuid: "68ebee93c4e844e19dc106d167ca7ee8" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 32879 } } peers { permanent_uuid: "d0897e4e6b774803b14a1609c99b2594" member_type: VOTER last_known_addr { host: "127.5.83.60" port: 36219 } } }
I20260808 04:09:39.284884 6032 sys_catalog.cc:458] T 00000000000000000000000000000000 P adc7645fa8bc478b9e11d8df76225fdf [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:09:39.285590 5815 sys_catalog.cc:564] T 00000000000000000000000000000000 P adc7645fa8bc478b9e11d8df76225fdf [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:09:39.285771 5780 raft_consensus.cc:482] T 00000000000000000000000000000000 P adc7645fa8bc478b9e11d8df76225fdf [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 04:09:39.285981 5780 raft_consensus.cc:3037] T 00000000000000000000000000000000 P adc7645fa8bc478b9e11d8df76225fdf [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:09:39.289276 5780 raft_consensus.cc:504] T 00000000000000000000000000000000 P adc7645fa8bc478b9e11d8df76225fdf [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc7645fa8bc478b9e11d8df76225fdf" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 45311 } } peers { permanent_uuid: "68ebee93c4e844e19dc106d167ca7ee8" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 32879 } } peers { permanent_uuid: "d0897e4e6b774803b14a1609c99b2594" member_type: VOTER last_known_addr { host: "127.5.83.60" port: 36219 } }
I20260808 04:09:39.291582 5972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "adc7645fa8bc478b9e11d8df76225fdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d0897e4e6b774803b14a1609c99b2594"
I20260808 04:09:39.292057 5972 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0897e4e6b774803b14a1609c99b2594 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:09:39.295420 5972 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d0897e4e6b774803b14a1609c99b2594 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate adc7645fa8bc478b9e11d8df76225fdf in term 1.
I20260808 04:09:39.295998 5743 leader_election.cc:302] T 00000000000000000000000000000000 P adc7645fa8bc478b9e11d8df76225fdf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: adc7645fa8bc478b9e11d8df76225fdf, d0897e4e6b774803b14a1609c99b2594; no voters:
I20260808 04:09:39.296463 6032 raft_consensus.cc:2781] T 00000000000000000000000000000000 P adc7645fa8bc478b9e11d8df76225fdf [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:09:39.297111 6032 raft_consensus.cc:686] T 00000000000000000000000000000000 P adc7645fa8bc478b9e11d8df76225fdf [term 1 LEADER]: Becoming Leader. State: Replica: adc7645fa8bc478b9e11d8df76225fdf, State: Running, Role: LEADER
I20260808 04:09:39.297629 6032 consensus_queue.cc:237] T 00000000000000000000000000000000 P adc7645fa8bc478b9e11d8df76225fdf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc7645fa8bc478b9e11d8df76225fdf" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 45311 } } peers { permanent_uuid: "68ebee93c4e844e19dc106d167ca7ee8" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 32879 } } peers { permanent_uuid: "d0897e4e6b774803b14a1609c99b2594" member_type: VOTER last_known_addr { host: "127.5.83.60" port: 36219 } }
I20260808 04:09:39.299697 5780 leader_election.cc:288] T 00000000000000000000000000000000 P adc7645fa8bc478b9e11d8df76225fdf [CANDIDATE]: Term 1 election: Requested vote from peers 68ebee93c4e844e19dc106d167ca7ee8 (127.5.83.61:32879), d0897e4e6b774803b14a1609c99b2594 (127.5.83.60:36219)
I20260808 04:09:39.301512 6035 sys_catalog.cc:455] T 00000000000000000000000000000000 P adc7645fa8bc478b9e11d8df76225fdf [sys.catalog]: SysCatalogTable state changed. Reason: New leader adc7645fa8bc478b9e11d8df76225fdf. Latest consensus state: current_term: 1 leader_uuid: "adc7645fa8bc478b9e11d8df76225fdf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc7645fa8bc478b9e11d8df76225fdf" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 45311 } } peers { permanent_uuid: "68ebee93c4e844e19dc106d167ca7ee8" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 32879 } } peers { permanent_uuid: "d0897e4e6b774803b14a1609c99b2594" member_type: VOTER last_known_addr { host: "127.5.83.60" port: 36219 } } }
I20260808 04:09:39.301887 6035 sys_catalog.cc:458] T 00000000000000000000000000000000 P adc7645fa8bc478b9e11d8df76225fdf [sys.catalog]: This master's current role is: LEADER
I20260808 04:09:39.304816 6039 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:09:39.322256 5910 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.83.60" port: 36219 } attempt: 5
I20260808 04:09:39.338755 6039 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260808 04:09:39.338968 5944 tablet.cc:2369] T 00000000000000000000000000000000 P d0897e4e6b774803b14a1609c99b2594: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:09:39.347681 5915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:39.304877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49574 (local address 127.5.83.61:32879)
0808 04:09:39.305066 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:39.305072 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:39.308073 (+ 3001us) server_negotiation.cc:407] Connection header received
0808 04:09:39.308443 (+ 370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:39.308449 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:39.308536 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:39.308609 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:39.312708 (+ 4099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:39.313948 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:39.317662 (+ 3714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:39.318315 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:39.318420 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:39.324250 (+ 5830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:09:39.324257 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:09:39.324262 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:09:39.324305 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:09:39.327917 (+ 3612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:09:39.340897 (+ 12980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:09:39.340915 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:09:39.340919 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:09:39.341083 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:09:39.343430 (+ 2347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:09:39.343436 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:09:39.343439 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:09:39.343858 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:09:39.344101 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:39.347081 (+ 2980us) server_negotiation.cc:299] Negotiation successful
0808 04:09:39.347325 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260808 04:09:39.355876 5748 tablet.cc:2369] T 00000000000000000000000000000000 P adc7645fa8bc478b9e11d8df76225fdf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:09:39.367868 6008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:39.305567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34066 (local address 127.5.83.60:36219)
0808 04:09:39.305730 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:39.305736 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:39.320163 (+ 14427us) server_negotiation.cc:407] Connection header received
0808 04:09:39.320657 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:39.320663 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:39.320747 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:39.320819 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:39.326802 (+ 5983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:39.327976 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:39.333049 (+ 5073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:39.333762 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:39.333867 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:39.348267 (+ 14400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:09:39.348274 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:09:39.348283 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:09:39.348326 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:09:39.354205 (+ 5879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:09:39.358704 (+ 4499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:09:39.358711 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:09:39.358715 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:09:39.358892 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:09:39.364154 (+ 5262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:09:39.364160 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:09:39.364164 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:09:39.364525 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:09:39.364793 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:39.367233 (+ 2440us) server_negotiation.cc:299] Negotiation successful
0808 04:09:39.367462 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 04:09:39.371055 6010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:39.312291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51612 (local address 127.5.83.62:45311)
0808 04:09:39.312454 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:39.312459 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:39.324639 (+ 12180us) server_negotiation.cc:407] Connection header received
0808 04:09:39.324866 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:39.324871 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:39.324981 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:39.325048 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:39.329825 (+ 4777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:39.330872 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:39.340786 (+ 9914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:39.341477 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:39.341572 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:39.353562 (+ 11990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:09:39.353569 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:09:39.353574 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:09:39.353620 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:09:39.357326 (+ 3706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:09:39.364963 (+ 7637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:09:39.364968 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:09:39.364971 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:09:39.365154 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:09:39.366103 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:09:39.366109 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:09:39.366113 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:09:39.366457 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:09:39.366686 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:39.370484 (+ 3798us) server_negotiation.cc:299] Negotiation successful
0808 04:09:39.370708 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 04:09:39.374018 5972 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d0897e4e6b774803b14a1609c99b2594 [term 1 FOLLOWER]: Refusing update from remote peer adc7645fa8bc478b9e11d8df76225fdf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:09:39.375168 6035 consensus_queue.cc:1035] T 00000000000000000000000000000000 P adc7645fa8bc478b9e11d8df76225fdf [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0897e4e6b774803b14a1609c99b2594" member_type: VOTER last_known_addr { host: "127.5.83.60" port: 36219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Aug 08 04:09:39 dist-test-slave-1l3l krb5kdc[5701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162178, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.61@KRBTEST.COM for kudu/127.5.83.60@KRBTEST.COM
I20260808 04:09:39.389917 6048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:39.323510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34072 (local address 127.5.83.60:36219)
0808 04:09:39.359222 (+ 35712us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:39.359228 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:39.359253 (+ 25us) server_negotiation.cc:407] Connection header received
0808 04:09:39.359306 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:39.359312 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:39.359391 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:39.360529 (+ 1138us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:39.360796 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:39.362020 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:39.373537 (+ 11517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:39.374216 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:39.374952 (+ 736us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:39.383258 (+ 8306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:09:39.383270 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:09:39.383277 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:09:39.383320 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:09:39.386875 (+ 3555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:09:39.387688 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:09:39.387694 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:09:39.387698 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:09:39.387899 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:09:39.388306 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:09:39.388310 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:09:39.388314 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:09:39.388655 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:09:39.388788 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:39.389252 (+ 464us) server_negotiation.cc:299] Negotiation successful
0808 04:09:39.389481 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35571,"thread_start_us":29834,"threads_started":1}
I20260808 04:09:39.389947 5912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:39.322822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.60:36219 (local address 127.0.0.1:34072)
0808 04:09:39.323192 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0808 04:09:39.323211 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 04:09:39.323308 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:09:39.359595 (+ 36287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:09:39.359601 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:09:39.360157 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:09:39.360372 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:09:39.360383 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:09:39.371821 (+ 11438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:09:39.371827 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:09:39.373327 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:09:39.373342 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:09:39.374387 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:09:39.374394 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:09:39.374608 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:39.378026 (+ 3418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:09:39.378049 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:09:39.380764 (+ 2715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:09:39.387024 (+ 6260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:09:39.387030 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:09:39.387034 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:09:39.387586 (+ 552us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:09:39.388023 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:09:39.388029 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:09:39.388032 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:09:39.388210 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:09:39.388805 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:09:39.388810 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:09:39.388972 (+ 162us) client_negotiation.cc:769] Sending connection context
0808 04:09:39.389144 (+ 172us) client_negotiation.cc:240] Negotiation successful
0808 04:09:39.389574 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255}
I20260808 04:09:39.397594 5910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68ebee93c4e844e19dc106d167ca7ee8: Bootstrap starting.
I20260808 04:09:39.405026 5910 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68ebee93c4e844e19dc106d167ca7ee8: Neither blocks nor log segments found. Creating new log.
I20260808 04:09:39.407335 5910 log.cc:824] T 00000000000000000000000000000000 P 68ebee93c4e844e19dc106d167ca7ee8: Log is configured to *not* fsync() on all Append() calls
I20260808 04:09:39.417843 5910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68ebee93c4e844e19dc106d167ca7ee8: No bootstrap required, opened a new log
I20260808 04:09:39.420111 6012 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0897e4e6b774803b14a1609c99b2594 [sys.catalog]: SysCatalogTable state changed. Reason: New leader adc7645fa8bc478b9e11d8df76225fdf. Latest consensus state: current_term: 1 leader_uuid: "adc7645fa8bc478b9e11d8df76225fdf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc7645fa8bc478b9e11d8df76225fdf" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 45311 } } peers { permanent_uuid: "68ebee93c4e844e19dc106d167ca7ee8" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 32879 } } peers { permanent_uuid: "d0897e4e6b774803b14a1609c99b2594" member_type: VOTER last_known_addr { host: "127.5.83.60" port: 36219 } } }
I20260808 04:09:39.420415 6012 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0897e4e6b774803b14a1609c99b2594 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:09:39.423477 6053 mvcc.cc:205] Tried to move back new op lower bound from 7316120286700793856 to 7316120286407720960. Current Snapshot: MvccSnapshot[applied={T|T < 7316120286700793856}]
I20260808 04:09:39.431190 6055 mvcc.cc:205] Tried to move back new op lower bound from 7316120286700793856 to 7316120286407720960. Current Snapshot: MvccSnapshot[applied={T|T < 7316120286700793856}]
I20260808 04:09:39.436501 5910 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68ebee93c4e844e19dc106d167ca7ee8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc7645fa8bc478b9e11d8df76225fdf" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 45311 } } peers { permanent_uuid: "68ebee93c4e844e19dc106d167ca7ee8" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 32879 } } peers { permanent_uuid: "d0897e4e6b774803b14a1609c99b2594" member_type: VOTER last_known_addr { host: "127.5.83.60" port: 36219 } }
I20260808 04:09:39.437175 5910 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68ebee93c4e844e19dc106d167ca7ee8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68ebee93c4e844e19dc106d167ca7ee8, State: Initialized, Role: FOLLOWER
I20260808 04:09:39.437346 6035 sys_catalog.cc:455] T 00000000000000000000000000000000 P adc7645fa8bc478b9e11d8df76225fdf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "adc7645fa8bc478b9e11d8df76225fdf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc7645fa8bc478b9e11d8df76225fdf" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 45311 } } peers { permanent_uuid: "68ebee93c4e844e19dc106d167ca7ee8" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 32879 } } peers { permanent_uuid: "d0897e4e6b774803b14a1609c99b2594" member_type: VOTER last_known_addr { host: "127.5.83.60" port: 36219 } } }
I20260808 04:09:39.437629 6035 sys_catalog.cc:458] T 00000000000000000000000000000000 P adc7645fa8bc478b9e11d8df76225fdf [sys.catalog]: This master's current role is: LEADER
I20260808 04:09:39.437568 6012 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0897e4e6b774803b14a1609c99b2594 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "adc7645fa8bc478b9e11d8df76225fdf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc7645fa8bc478b9e11d8df76225fdf" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 45311 } } peers { permanent_uuid: "68ebee93c4e844e19dc106d167ca7ee8" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 32879 } } peers { permanent_uuid: "d0897e4e6b774803b14a1609c99b2594" member_type: VOTER last_known_addr { host: "127.5.83.60" port: 36219 } } }
I20260808 04:09:39.437773 6012 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0897e4e6b774803b14a1609c99b2594 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:09:39.438392 6039 catalog_manager.cc:1347] Generated new cluster ID: 59b8c82c01384962ab6d1fdbe5661042
I20260808 04:09:39.438509 6039 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:09:39.438041 5910 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68ebee93c4e844e19dc106d167ca7ee8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc7645fa8bc478b9e11d8df76225fdf" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 45311 } } peers { permanent_uuid: "68ebee93c4e844e19dc106d167ca7ee8" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 32879 } } peers { permanent_uuid: "d0897e4e6b774803b14a1609c99b2594" member_type: VOTER last_known_addr { host: "127.5.83.60" port: 36219 } }
I20260808 04:09:39.457860 5875 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68ebee93c4e844e19dc106d167ca7ee8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:09:39.461439 5875 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 68ebee93c4e844e19dc106d167ca7ee8 [term 1 FOLLOWER]: Refusing update from remote peer adc7645fa8bc478b9e11d8df76225fdf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:09:39.462060 5876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "adc7645fa8bc478b9e11d8df76225fdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "68ebee93c4e844e19dc106d167ca7ee8"
I20260808 04:09:39.465320 5876 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 68ebee93c4e844e19dc106d167ca7ee8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate adc7645fa8bc478b9e11d8df76225fdf in term 1.
I20260808 04:09:39.466377 6032 consensus_queue.cc:1035] T 00000000000000000000000000000000 P adc7645fa8bc478b9e11d8df76225fdf [LEADER]: Connected to new peer: Peer: permanent_uuid: "68ebee93c4e844e19dc106d167ca7ee8" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 32879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:09:39.467392 5910 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68ebee93c4e844e19dc106d167ca7ee8 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:09:39.484615 6071 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260808 04:09:39.490789 5718 master_runner.cc:186] Error getting master registration for 127.5.83.61:32879: OK, instance_id { permanent_uuid: "68ebee93c4e844e19dc106d167ca7ee8" instance_seqno: 1786162178602111 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 04:09:39.504650 5718 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 04:09:39.504833 6039 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:09:39.506544 6039 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:09:39.522046 6039 catalog_manager.cc:5868] T 00000000000000000000000000000000 P adc7645fa8bc478b9e11d8df76225fdf: Generated new TSK 0
I20260808 04:09:39.522926 6039 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:09:39.544485 6067 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68ebee93c4e844e19dc106d167ca7ee8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader adc7645fa8bc478b9e11d8df76225fdf. Latest consensus state: current_term: 1 leader_uuid: "adc7645fa8bc478b9e11d8df76225fdf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc7645fa8bc478b9e11d8df76225fdf" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 45311 } } peers { permanent_uuid: "68ebee93c4e844e19dc106d167ca7ee8" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 32879 } } peers { permanent_uuid: "d0897e4e6b774803b14a1609c99b2594" member_type: VOTER last_known_addr { host: "127.5.83.60" port: 36219 } } }
I20260808 04:09:39.544880 6067 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68ebee93c4e844e19dc106d167ca7ee8 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:09:39.545701 6066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68ebee93c4e844e19dc106d167ca7ee8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "adc7645fa8bc478b9e11d8df76225fdf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc7645fa8bc478b9e11d8df76225fdf" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 45311 } } peers { permanent_uuid: "68ebee93c4e844e19dc106d167ca7ee8" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 32879 } } peers { permanent_uuid: "d0897e4e6b774803b14a1609c99b2594" member_type: VOTER last_known_addr { host: "127.5.83.60" port: 36219 } } }
I20260808 04:09:39.546072 6066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68ebee93c4e844e19dc106d167ca7ee8 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:09:39.546279 6086 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68ebee93c4e844e19dc106d167ca7ee8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "adc7645fa8bc478b9e11d8df76225fdf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc7645fa8bc478b9e11d8df76225fdf" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 45311 } } peers { permanent_uuid: "68ebee93c4e844e19dc106d167ca7ee8" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 32879 } } peers { permanent_uuid: "d0897e4e6b774803b14a1609c99b2594" member_type: VOTER last_known_addr { host: "127.5.83.60" port: 36219 } } }
I20260808 04:09:39.546599 6086 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68ebee93c4e844e19dc106d167ca7ee8 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:09:39.546602 6032 sys_catalog.cc:455] T 00000000000000000000000000000000 P adc7645fa8bc478b9e11d8df76225fdf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "adc7645fa8bc478b9e11d8df76225fdf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc7645fa8bc478b9e11d8df76225fdf" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 45311 } } peers { permanent_uuid: "68ebee93c4e844e19dc106d167ca7ee8" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 32879 } } peers { permanent_uuid: "d0897e4e6b774803b14a1609c99b2594" member_type: VOTER last_known_addr { host: "127.5.83.60" port: 36219 } } }
I20260808 04:09:39.546885 6032 sys_catalog.cc:458] T 00000000000000000000000000000000 P adc7645fa8bc478b9e11d8df76225fdf [sys.catalog]: This master's current role is: LEADER
I20260808 04:09:39.574638 6088 catalog_manager.cc:1259] Loaded cluster ID: 59b8c82c01384962ab6d1fdbe5661042
I20260808 04:09:39.574753 6088 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 68ebee93c4e844e19dc106d167ca7ee8: loading cluster ID for follower catalog manager: success
I20260808 04:09:39.578629 6088 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 68ebee93c4e844e19dc106d167ca7ee8: acquiring CA information for follower catalog manager: success
I20260808 04:09:39.581090 6088 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 68ebee93c4e844e19dc106d167ca7ee8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260808 04:09:39.684643 5915 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:09:39.660003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49588 (local address 127.5.83.61:32879)
0808 04:09:39.660192 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:39.660200 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:39.681393 (+ 21193us) server_negotiation.cc:407] Connection header received
0808 04:09:39.682965 (+ 1572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:39.682972 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:39.683084 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:39.683162 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:39.684389 (+ 1227us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49588: BlockingRecv error: recv got EOF from 127.0.0.1:49588 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260808 04:09:39.687363 6048 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:09:39.666395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34088 (local address 127.5.83.60:36219)
0808 04:09:39.666576 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:39.666583 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:39.685410 (+ 18827us) server_negotiation.cc:407] Connection header received
0808 04:09:39.685823 (+ 413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:39.685829 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:39.685936 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:39.686014 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:39.687104 (+ 1090us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34088: BlockingRecv error: recv got EOF from 127.0.0.1:34088 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260808 04:09:39.690248 6010 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:09:39.677075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51618 (local address 127.5.83.62:45311)
0808 04:09:39.677253 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:39.677259 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:39.688177 (+ 10918us) server_negotiation.cc:407] Connection header received
0808 04:09:39.688767 (+ 590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:39.688774 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:39.688857 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:39.688932 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:39.689977 (+ 1045us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51618: BlockingRecv error: recv got EOF from 127.0.0.1:51618 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 04:09:39.739421 5915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:39.720666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49602 (local address 127.5.83.61:32879)
0808 04:09:39.720845 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:39.720852 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:39.731449 (+ 10597us) server_negotiation.cc:407] Connection header received
0808 04:09:39.731994 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:39.732000 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:39.732219 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:39.732374 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:09:39.733082 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:39.734274 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:39.736407 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:39.737066 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:39.737152 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:39.737917 (+ 765us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:09:39.738593 (+ 676us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:09:39.738708 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:39.738953 (+ 245us) server_negotiation.cc:299] Negotiation successful
0808 04:09:39.739053 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260808 04:09:39.765993 6048 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:09:39.762981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34104 (local address 127.5.83.60:36219)
0808 04:09:39.763157 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:39.763163 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:39.764278 (+ 1115us) server_negotiation.cc:407] Connection header received
0808 04:09:39.764574 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:39.764581 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:39.764706 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:39.764828 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:39.765726 (+ 898us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34104: BlockingRecv error: recv got EOF from 127.0.0.1:34104 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 04:09:39.783695 5915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:39.776546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49608 (local address 127.5.83.61:32879)
0808 04:09:39.776725 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:39.776731 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:39.777021 (+ 290us) server_negotiation.cc:407] Connection header received
0808 04:09:39.777301 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:39.777306 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:39.777446 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:39.777513 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:09:39.778150 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:39.779339 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:39.781396 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:39.782084 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:39.782192 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:39.782808 (+ 616us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:09:39.782983 (+ 175us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:09:39.783045 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:39.783279 (+ 234us) server_negotiation.cc:299] Negotiation successful
0808 04:09:39.783374 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260808 04:09:39.794051 6048 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:09:39.790971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34112 (local address 127.5.83.60:36219)
0808 04:09:39.791146 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:39.791151 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:39.792249 (+ 1098us) server_negotiation.cc:407] Connection header received
0808 04:09:39.792581 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:39.792588 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:39.792704 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:39.792776 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:39.793840 (+ 1064us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34112: BlockingRecv error: recv got EOF from 127.0.0.1:34112 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 04:09:39.807983 5915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:39.801140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49618 (local address 127.5.83.61:32879)
0808 04:09:39.801343 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:39.801349 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:39.801892 (+ 543us) server_negotiation.cc:407] Connection header received
0808 04:09:39.802133 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:39.802138 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:39.802229 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:39.802334 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:09:39.802787 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:39.803931 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:39.805859 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:39.806552 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:39.806682 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:39.807139 (+ 457us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:09:39.807275 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:09:39.807363 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:39.807528 (+ 165us) server_negotiation.cc:299] Negotiation successful
0808 04:09:39.807611 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260808 04:09:39.822882 6048 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:09:39.820535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34120 (local address 127.5.83.60:36219)
0808 04:09:39.820704 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:39.820710 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:39.821205 (+ 495us) server_negotiation.cc:407] Connection header received
0808 04:09:39.821495 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:39.821500 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:39.821604 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:39.821671 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:39.822633 (+ 962us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34120: BlockingRecv error: recv got EOF from 127.0.0.1:34120 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 04:09:39.837494 5915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:39.830693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49632 (local address 127.5.83.61:32879)
0808 04:09:39.830916 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:39.830922 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:39.831371 (+ 449us) server_negotiation.cc:407] Connection header received
0808 04:09:39.831537 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:39.831543 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:39.831638 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:39.831806 (+ 168us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:09:39.832240 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:39.833428 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:39.835294 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:39.836060 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:39.836206 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:39.836688 (+ 482us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:09:39.836821 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:09:39.836973 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:39.837129 (+ 156us) server_negotiation.cc:299] Negotiation successful
0808 04:09:39.837213 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
W20260808 04:09:39.853778 6048 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:09:39.848335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34130 (local address 127.5.83.60:36219)
0808 04:09:39.848614 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:39.848620 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:39.852286 (+ 3666us) server_negotiation.cc:407] Connection header received
0808 04:09:39.852479 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:39.852484 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:39.852578 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:39.852689 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:39.853553 (+ 864us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34130: BlockingRecv error: recv got EOF from 127.0.0.1:34130 (error 108)
Metrics: {"server-negotiator.queue_time_us":162}
I20260808 04:09:39.871986 5915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:39.864975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49642 (local address 127.5.83.61:32879)
0808 04:09:39.865164 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:39.865170 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:39.865670 (+ 500us) server_negotiation.cc:407] Connection header received
0808 04:09:39.865875 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:39.865881 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:39.865968 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:39.866059 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:09:39.866515 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:39.867731 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:39.869812 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:39.870490 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:39.870634 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:39.871122 (+ 488us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:09:39.871261 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:09:39.871356 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:39.871537 (+ 181us) server_negotiation.cc:299] Negotiation successful
0808 04:09:39.871620 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260808 04:09:39.887918 6048 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:09:39.885444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34132 (local address 127.5.83.60:36219)
0808 04:09:39.885629 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:39.885636 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:39.886222 (+ 586us) server_negotiation.cc:407] Connection header received
0808 04:09:39.886533 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:39.886539 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:39.886640 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:39.886752 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:39.887653 (+ 901us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34132: BlockingRecv error: recv got EOF from 127.0.0.1:34132 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 04:09:39.914095 5915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:39.907307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49648 (local address 127.5.83.61:32879)
0808 04:09:39.907494 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:39.907501 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:39.908033 (+ 532us) server_negotiation.cc:407] Connection header received
0808 04:09:39.908223 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:39.908230 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:39.908320 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:39.908433 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:09:39.908923 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:39.910167 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:39.912009 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:39.912648 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:39.912810 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:39.913288 (+ 478us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:09:39.913431 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:09:39.913531 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:39.913706 (+ 175us) server_negotiation.cc:299] Negotiation successful
0808 04:09:39.913799 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260808 04:09:39.927420 6048 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:09:39.925287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34138 (local address 127.5.83.60:36219)
0808 04:09:39.925456 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:39.925462 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:39.925950 (+ 488us) server_negotiation.cc:407] Connection header received
0808 04:09:39.926091 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:39.926097 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:39.926198 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:39.926314 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:39.927185 (+ 871us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34138: BlockingRecv error: recv got EOF from 127.0.0.1:34138 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 04:09:39.968629 5915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:39.962532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49654 (local address 127.5.83.61:32879)
0808 04:09:39.962749 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:39.962755 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:39.963074 (+ 319us) server_negotiation.cc:407] Connection header received
0808 04:09:39.963235 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:39.963242 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:39.963335 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:39.963450 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:09:39.963906 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:39.965168 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:39.966485 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:39.967240 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:39.967380 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:39.967766 (+ 386us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:09:39.967919 (+ 153us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:09:39.968090 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:39.968191 (+ 101us) server_negotiation.cc:299] Negotiation successful
0808 04:09:39.968273 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260808 04:09:39.976394 6048 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:09:39.974249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34152 (local address 127.5.83.60:36219)
0808 04:09:39.974472 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:39.974478 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:39.974874 (+ 396us) server_negotiation.cc:407] Connection header received
0808 04:09:39.975084 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:39.975088 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:39.975170 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:39.975251 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:39.976158 (+ 907us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34152: BlockingRecv error: recv got EOF from 127.0.0.1:34152 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260808 04:09:40.048753 5915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:40.043674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49662 (local address 127.5.83.61:32879)
0808 04:09:40.043889 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:40.043895 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:40.044269 (+ 374us) server_negotiation.cc:407] Connection header received
0808 04:09:40.044443 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:40.044447 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:40.044516 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:40.044596 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:09:40.045026 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:40.045747 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:40.047267 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:40.047662 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:40.047798 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:40.048189 (+ 391us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:09:40.048297 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:09:40.048360 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:40.048485 (+ 125us) server_negotiation.cc:299] Negotiation successful
0808 04:09:40.048547 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
W20260808 04:09:40.055353 6048 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:09:40.053711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34162 (local address 127.5.83.60:36219)
0808 04:09:40.053853 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:40.053857 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:40.054198 (+ 341us) server_negotiation.cc:407] Connection header received
0808 04:09:40.054325 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:40.054328 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:40.054398 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:40.054505 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:40.055179 (+ 674us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34162: BlockingRecv error: recv got EOF from 127.0.0.1:34162 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 04:09:40.191984 5915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:40.186971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49668 (local address 127.5.83.61:32879)
0808 04:09:40.187213 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:40.187219 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:40.187508 (+ 289us) server_negotiation.cc:407] Connection header received
0808 04:09:40.187675 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:40.187679 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:40.187781 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:40.187861 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:09:40.188284 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:40.188973 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:40.190404 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:40.190808 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:40.190908 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:40.191313 (+ 405us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:09:40.191418 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:09:40.191477 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:40.191605 (+ 128us) server_negotiation.cc:299] Negotiation successful
0808 04:09:40.191664 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260808 04:09:40.198879 6048 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:09:40.197088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34170 (local address 127.5.83.60:36219)
0808 04:09:40.197328 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:40.197334 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:40.197537 (+ 203us) server_negotiation.cc:407] Connection header received
0808 04:09:40.197705 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:40.197709 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:40.197780 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:40.197871 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:40.198626 (+ 755us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34170: BlockingRecv error: recv got EOF from 127.0.0.1:34170 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260808 04:09:40.200558 6027 catalog_manager.cc:1259] Loaded cluster ID: 59b8c82c01384962ab6d1fdbe5661042
I20260808 04:09:40.200621 6027 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d0897e4e6b774803b14a1609c99b2594: loading cluster ID for follower catalog manager: success
I20260808 04:09:40.203061 6027 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d0897e4e6b774803b14a1609c99b2594: acquiring CA information for follower catalog manager: success
I20260808 04:09:40.204768 6027 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d0897e4e6b774803b14a1609c99b2594: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 04:09:40.463550 5915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:40.458617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49684 (local address 127.5.83.61:32879)
0808 04:09:40.458848 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:40.458855 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:40.459001 (+ 146us) server_negotiation.cc:407] Connection header received
0808 04:09:40.459216 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:40.459220 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:40.459291 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:40.459383 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:09:40.459863 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:40.460600 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:40.461914 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:40.462330 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:40.462459 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:40.462862 (+ 403us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:09:40.462982 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:09:40.463060 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:40.463209 (+ 149us) server_negotiation.cc:299] Negotiation successful
0808 04:09:40.463292 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260808 04:09:40.473639 6048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:40.468814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34172 (local address 127.5.83.60:36219)
0808 04:09:40.469022 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:40.469028 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:40.469195 (+ 167us) server_negotiation.cc:407] Connection header received
0808 04:09:40.469343 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:40.469346 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:40.469503 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:40.469590 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:09:40.469923 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:40.470597 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:40.471816 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:40.472210 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:40.472298 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:40.472700 (+ 402us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:09:40.473190 (+ 490us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:09:40.473294 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:40.473331 (+ 37us) server_negotiation.cc:299] Negotiation successful
0808 04:09:40.473389 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260808 04:09:40.497191 6247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:40.479113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51628 (local address 127.5.83.62:45311)
0808 04:09:40.490379 (+ 11266us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:40.490384 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:40.490512 (+ 128us) server_negotiation.cc:407] Connection header received
0808 04:09:40.491030 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:40.491036 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:40.491265 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:40.491463 (+ 198us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:09:40.492263 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:40.493317 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:40.494963 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:40.495369 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:40.495480 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:40.496023 (+ 543us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:09:40.496448 (+ 425us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:09:40.496566 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:40.496717 (+ 151us) server_negotiation.cc:299] Negotiation successful
0808 04:09:40.496802 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11127,"thread_start_us":11426,"threads_started":1}
I20260808 04:09:40.497887 6048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:40.479922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34176 (local address 127.5.83.60:36219)
0808 04:09:40.480224 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:40.480230 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:40.491266 (+ 11036us) server_negotiation.cc:407] Connection header received
0808 04:09:40.491494 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:40.491499 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:40.491583 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:40.491645 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:09:40.492614 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:40.493682 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:40.496114 (+ 2432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:40.496486 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:40.496598 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:40.497135 (+ 537us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:09:40.497242 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:09:40.497322 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:40.497543 (+ 221us) server_negotiation.cc:299] Negotiation successful
0808 04:09:40.497627 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203}
I20260808 04:09:40.498656 5915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:40.479696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49686 (local address 127.5.83.61:32879)
0808 04:09:40.480104 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:40.480110 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:40.490958 (+ 10848us) server_negotiation.cc:407] Connection header received
0808 04:09:40.491221 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:40.491226 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:40.491305 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:40.491399 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:09:40.492722 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:40.493768 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:40.496058 (+ 2290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:40.496665 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:40.496780 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:40.497362 (+ 582us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:09:40.497484 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:09:40.497561 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:40.498335 (+ 774us) server_negotiation.cc:299] Negotiation successful
0808 04:09:40.498430 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255}
I20260808 04:09:40.502892 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 5718
I20260808 04:09:40.503892 5809 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:09:40.612188 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 5718
I20260808 04:09:40.636029 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 5823
I20260808 04:09:40.637184 5905 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:09:40.743579 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 5823
I20260808 04:09:40.766426 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 5919
I20260808 04:09:40.767536 5995 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:09:40.873390 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 5919
2026-08-08T04:09:40Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4696 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161826605655-5452-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:09:40 dist-test-slave-1l3l krb5kdc[6260](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:09:40 dist-test-slave-1l3l krb5kdc[6260](info): set up 2 sockets
Aug 08 04:09:40 dist-test-slave-1l3l krb5kdc[6260](info): commencing operation
krb5kdc: starting...
W20260808 04:09:42.101976 5452 mini_kdc.cc:121] Time spent starting KDC: real 1.186s user 0.001s sys 0.042s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161826605655-5452-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161826605655-5452-0/krb5kdc/test-user.keytab.
Aug 08 04:09:42 dist-test-slave-1l3l krb5kdc[6260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:09:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:09:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.83.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab.
Entry for principal kudu/127.5.83.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab.
I20260808 04:09:42.383756 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161826605655-5452-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161826605655-5452-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161826605655-5452-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:40575
--webserver_interface=127.5.83.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab
--principal=kudu/127.5.83.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:41623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.62:40575,127.5.83.61:43419,127.5.83.60:40519
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:09:42.706182 6276 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 04:09:42.706470 6276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:09:42.706532 6276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:09:42.706601 6276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:09:42.713590 6276 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:09:42.713668 6276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:09:42.713737 6276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:09:42.713804 6276 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:09:42.713887 6276 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:09:42.723614 6276 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:41623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161826605655-5452-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161826605655-5452-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.62:40575,127.5.83.61:43419,127.5.83.60:40519
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.62.keytab
--principal=kudu/127.5.83.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.62:40575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161826605655-5452-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:09:42.724788 6276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:09:42.725999 6276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:09:42.737046 6284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:09:42.737111 6282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:09:42.737282 6281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:09:42.737854 6276 server_base.cc:1034] running on GCE node
I20260808 04:09:42.738579 6276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:09:42.739880 6276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:09:42.741097 6276 hybrid_clock.cc:648] HybridClock initialized: now 1786162182741044 us; error 27 us; skew 500 ppm
I20260808 04:09:42.741376 6276 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:09:42 dist-test-slave-1l3l krb5kdc[6260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162182, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:09:42.747047 6276 init.cc:377] Logged in from keytab as kudu/127.5.83.62@KRBTEST.COM (short username kudu)
I20260808 04:09:42.748517 6276 webserver.cc:460] Webserver started at http://127.5.83.62:45913/ using document root <none> and password file <none>
I20260808 04:09:42.749147 6276 fs_manager.cc:359] Metadata directory not provided
I20260808 04:09:42.749231 6276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:09:42.749495 6276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:09:42.752329 6276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161826605655-5452-0/minicluster-data/master-0/data/instance:
uuid: "9dcb00ee5483403f89c52903124ef8ba"
format_stamp: "Formatted at 2026-08-08 04:09:42 on dist-test-slave-1l3l"
server_key: "31c4442b1984b340c27269c87c570263"
server_key_iv: "8c31d39a4c641ce905b07656d153f9d4"
server_key_version: "encryptionkey@0"
I20260808 04:09:42.753049 6276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161826605655-5452-0/minicluster-data/master-0/wal/instance:
uuid: "9dcb00ee5483403f89c52903124ef8ba"
format_stamp: "Formatted at 2026-08-08 04:09:42 on dist-test-slave-1l3l"
server_key: "31c4442b1984b340c27269c87c570263"
server_key_iv: "8c31d39a4c641ce905b07656d153f9d4"
server_key_version: "encryptionkey@0"
I20260808 04:09:42.757767 6276 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 04:09:42.760818 6296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:09:42.762001 6276 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:09:42.762138 6276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161826605655-5452-0/minicluster-data/master-0/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161826605655-5452-0/minicluster-data/master-0/wal
uuid: "9dcb00ee5483403f89c52903124ef8ba"
format_stamp: "Formatted at 2026-08-08 04:09:42 on dist-test-slave-1l3l"
server_key: "31c4442b1984b340c27269c87c570263"
server_key_iv: "8c31d39a4c641ce905b07656d153f9d4"
server_key_version: "encryptionkey@0"
I20260808 04:09:42.762297 6276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161826605655-5452-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161826605655-5452-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161826605655-5452-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:09:42.781996 6276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:09:42.785549 6276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:09:42.785761 6276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:09:42.802621 6276 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.62:40575
I20260808 04:09:42.802640 6366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.62:40575 every 8 connection(s)
I20260808 04:09:42.803860 6276 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161826605655-5452-0/minicluster-data/master-0/data/info.pb
I20260808 04:09:42.807289 6367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:09:42.809463 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 6276
I20260808 04:09:42.809649 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161826605655-5452-0/minicluster-data/master-0/wal/instance
I20260808 04:09:42.810200 5452 external_mini_cluster.cc:1428] Setting key 1bee6e0133ae996ae85843e2567d2849
I20260808 04:09:42.815124 6367 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.83.62" port: 40575 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 04:09:42 dist-test-slave-1l3l krb5kdc[6260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162182, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.62@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:09:42.836426 6370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:42.818646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40510 (local address 127.5.83.62:40575)
0808 04:09:42.820107 (+ 1461us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:42.820115 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:42.820163 (+ 48us) server_negotiation.cc:407] Connection header received
0808 04:09:42.821628 (+ 1465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:42.821633 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:42.821849 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:42.822027 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:42.823240 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:42.824551 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:42.825870 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:42.826391 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:42.826571 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:42.829797 (+ 3226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:09:42.829804 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:09:42.829815 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:09:42.829842 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:09:42.832406 (+ 2564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:09:42.833024 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:09:42.833029 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:09:42.833032 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:09:42.833170 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:09:42.833527 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:09:42.833532 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:09:42.833534 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:09:42.833993 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:09:42.834175 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:42.834694 (+ 519us) server_negotiation.cc:299] Negotiation successful
0808 04:09:42.834979 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":1276,"thread_start_us":1129,"threads_started":1}
I20260808 04:09:42.836469 6369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:42.818495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:40575 (local address 127.0.0.1:40510)
0808 04:09:42.819321 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0808 04:09:42.819377 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0808 04:09:42.821287 (+ 1910us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:09:42.822065 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:09:42.822071 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:09:42.822733 (+ 662us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:09:42.823048 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:09:42.823064 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:09:42.824693 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:09:42.824697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:09:42.825735 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:09:42.825748 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:09:42.826612 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:09:42.826619 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:09:42.826866 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:42.827496 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:09:42.827518 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:09:42.829597 (+ 2079us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:09:42.832521 (+ 2924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:09:42.832532 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:09:42.832543 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:09:42.832908 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:09:42.833272 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:09:42.833277 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:09:42.833280 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:09:42.833431 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:09:42.834128 (+ 697us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:09:42.834135 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:09:42.834341 (+ 206us) client_negotiation.cc:769] Sending connection context
0808 04:09:42.834570 (+ 229us) client_negotiation.cc:240] Negotiation successful
0808 04:09:42.835103 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"mutex_wait_us":167,"thread_start_us":372,"threads_started":1}
I20260808 04:09:42.839171 6367 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.83.61" port: 43419 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.5.83.61@KRBTEST.COM; defaulting to no policy
I20260808 04:09:42.840153 6369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:42.839796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.61:43419 (local address 127.0.0.1:53234)
0808 04:09:42.839943 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0808 04:09:42.840045 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.61:43419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260808 04:09:42.840361 6303 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.83.61:43419: connect: Connection refused (error 111)
I20260808 04:09:42.841759 6369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:42.841373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.61:43419 (local address 127.0.0.1:53236)
0808 04:09:42.841515 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0808 04:09:42.841599 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.61:43419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260808 04:09:42.841955 6367 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.83.61:43419: Network error: Client connection negotiation failed: client connection to 127.5.83.61:43419: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.61.keytab.
Entry for principal kudu/127.5.83.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.61.keytab.
I20260808 04:09:42.872639 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161826605655-5452-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161826605655-5452-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161826605655-5452-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161826605655-5452-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.61:43419
--webserver_interface=127.5.83.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.61.keytab
--principal=kudu/127.5.83.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:41623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.62:40575,127.5.83.61:43419,127.5.83.60:40519
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:09:42.891280 6367 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.83.61" port: 43419 } attempt: 1
I20260808 04:09:42.892413 6369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:42.891991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.61:43419 (local address 127.0.0.1:53244)
0808 04:09:42.892177 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0808 04:09:42.892279 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.61:43419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260808 04:09:42.893436 6369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:42.893095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.61:43419 (local address 127.0.0.1:53258)
0808 04:09:42.893231 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0808 04:09:42.893303 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.61:43419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260808 04:09:42.893625 6367 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.83.61:43419: Network error: Client connection negotiation failed: client connection to 127.5.83.61:43419: connect: Connection refused (error 111)
I20260808 04:09:42.961892 6367 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.83.61" port: 43419 } attempt: 2
I20260808 04:09:42.963129 6369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:42.962637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.61:43419 (local address 127.0.0.1:53266)
0808 04:09:42.962892 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0808 04:09:42.962989 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.61:43419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260808 04:09:42.964143 6369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:42.963814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.61:43419 (local address 127.0.0.1:53274)
0808 04:09:42.963950 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0808 04:09:42.964019 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.61:43419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260808 04:09:42.964347 6367 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.83.61:43419: Network error: Client connection negotiation failed: client connection to 127.5.83.61:43419: connect: Connection refused (error 111)
I20260808 04:09:43.055572 6367 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.83.61" port: 43419 } attempt: 3
I20260808 04:09:43.056792 6369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:43.056359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.61:43419 (local address 127.0.0.1:53286)
0808 04:09:43.056549 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0808 04:09:43.056646 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.61:43419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260808 04:09:43.057873 6369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:43.057525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.61:43419 (local address 127.0.0.1:53292)
0808 04:09:43.057663 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0808 04:09:43.057734 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.61:43419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260808 04:09:43.058085 6367 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.83.61:43419: Network error: Client connection negotiation failed: client connection to 127.5.83.61:43419: connect: Connection refused (error 111)
W20260808 04:09:43.193637 6377 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 04:09:43.193926 6377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:09:43.193997 6377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:09:43.194077 6377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:09:43.201104 6377 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:09:43.201184 6377 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:09:43.201244 6377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:09:43.201395 6377 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
I20260808 04:09:43.201325 6367 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.83.61" port: 43419 } attempt: 4
W20260808 04:09:43.201502 6377 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:09:43.202466 6369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:43.201957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.61:43419 (local address 127.0.0.1:53304)
0808 04:09:43.202243 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0808 04:09:43.202333 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.61:43419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
I20260808 04:09:43.203454 6369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:43.203119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.61:43419 (local address 127.0.0.1:53314)
0808 04:09:43.203254 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0808 04:09:43.203336 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.61:43419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260808 04:09:43.203652 6367 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.83.61:43419: Network error: Client connection negotiation failed: client connection to 127.5.83.61:43419: connect: Connection refused (error 111)
I20260808 04:09:43.211341 6377 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:41623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161826605655-5452-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161826605655-5452-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.62:40575,127.5.83.61:43419,127.5.83.60:40519
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.61.keytab
--principal=kudu/127.5.83.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.61:43419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161826605655-5452-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.5.83.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161826605655-5452-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:09:43.212574 6377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:09:43.213791 6377 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:09:43.225111 6383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:09:43.225075 6387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:09:43.225147 6385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:09:43.225852 6377 server_base.cc:1034] running on GCE node
I20260808 04:09:43.226583 6377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:09:43.227902 6377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:09:43.229131 6377 hybrid_clock.cc:648] HybridClock initialized: now 1786162183229055 us; error 45 us; skew 500 ppm
I20260808 04:09:43.229444 6377 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:09:43 dist-test-slave-1l3l krb5kdc[6260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162183, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:09:43.232484 6377 init.cc:377] Logged in from keytab as kudu/127.5.83.61@KRBTEST.COM (short username kudu)
I20260808 04:09:43.233804 6377 webserver.cc:460] Webserver started at http://127.5.83.61:33027/ using document root <none> and password file <none>
I20260808 04:09:43.234406 6377 fs_manager.cc:359] Metadata directory not provided
I20260808 04:09:43.234498 6377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:09:43.234757 6377 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:09:43.237586 6377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161826605655-5452-0/minicluster-data/master-1/data/instance:
uuid: "6facd467610c48b4b5f053749ecd7b59"
format_stamp: "Formatted at 2026-08-08 04:09:43 on dist-test-slave-1l3l"
server_key: "b1f2a6db0b9b69529d917ec42d767ff1"
server_key_iv: "f5ed758dd8afa5eeed78ac256c798df5"
server_key_version: "encryptionkey@0"
I20260808 04:09:43.238278 6377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161826605655-5452-0/minicluster-data/master-1/wal/instance:
uuid: "6facd467610c48b4b5f053749ecd7b59"
format_stamp: "Formatted at 2026-08-08 04:09:43 on dist-test-slave-1l3l"
server_key: "b1f2a6db0b9b69529d917ec42d767ff1"
server_key_iv: "f5ed758dd8afa5eeed78ac256c798df5"
server_key_version: "encryptionkey@0"
I20260808 04:09:43.243062 6377 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 04:09:43.246209 6396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:09:43.247454 6377 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:09:43.247605 6377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161826605655-5452-0/minicluster-data/master-1/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161826605655-5452-0/minicluster-data/master-1/wal
uuid: "6facd467610c48b4b5f053749ecd7b59"
format_stamp: "Formatted at 2026-08-08 04:09:43 on dist-test-slave-1l3l"
server_key: "b1f2a6db0b9b69529d917ec42d767ff1"
server_key_iv: "f5ed758dd8afa5eeed78ac256c798df5"
server_key_version: "encryptionkey@0"
I20260808 04:09:43.247828 6377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161826605655-5452-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161826605655-5452-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161826605655-5452-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:09:43.269735 6377 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:09:43.273388 6377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:09:43.273634 6377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:09:43.290938 6377 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.61:43419
I20260808 04:09:43.290982 6471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.61:43419 every 8 connection(s)
I20260808 04:09:43.292147 6377 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161826605655-5452-0/minicluster-data/master-1/data/info.pb
I20260808 04:09:43.295523 6472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:09:43.297336 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 6377
I20260808 04:09:43.297525 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161826605655-5452-0/minicluster-data/master-1/wal/instance
I20260808 04:09:43.298115 5452 external_mini_cluster.cc:1428] Setting key 9bd88cf121b14378b7bb54ee075c55db
I20260808 04:09:43.303486 6472 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.83.62" port: 40575 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 04:09:43 dist-test-slave-1l3l krb5kdc[6260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162183, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.61@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:09:43.323547 6370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:43.306282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40522 (local address 127.5.83.62:40575)
0808 04:09:43.306483 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:43.306489 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:43.307623 (+ 1134us) server_negotiation.cc:407] Connection header received
0808 04:09:43.309443 (+ 1820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:43.309449 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:43.309523 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:43.309612 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:43.311005 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:43.311698 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:43.313771 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:43.314188 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:43.314298 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:43.317922 (+ 3624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:09:43.317927 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:09:43.317930 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:09:43.317962 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:09:43.320568 (+ 2606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:09:43.321422 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:09:43.321426 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:09:43.321428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:09:43.321540 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:09:43.322003 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:09:43.322006 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:09:43.322008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:09:43.322276 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:09:43.322405 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:43.323136 (+ 731us) server_negotiation.cc:299] Negotiation successful
0808 04:09:43.323297 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 04:09:43.324751 6474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:43.306700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:40575 (local address 127.0.0.1:40522)
0808 04:09:43.307458 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0808 04:09:43.307510 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 04:09:43.309178 (+ 1668us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:09:43.309838 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:09:43.309845 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:09:43.310341 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:09:43.310834 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:09:43.310851 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:09:43.311871 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:09:43.311875 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:09:43.313615 (+ 1740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:09:43.313631 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:09:43.314409 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:09:43.314417 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:09:43.314690 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:43.315363 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:09:43.315393 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:09:43.317706 (+ 2313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:09:43.320716 (+ 3010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:09:43.320726 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:09:43.320740 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:09:43.321316 (+ 576us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:09:43.321637 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:09:43.321642 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:09:43.321645 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:09:43.321871 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:09:43.322396 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:09:43.322403 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:09:43.322773 (+ 370us) client_negotiation.cc:769] Sending connection context
0808 04:09:43.323044 (+ 271us) client_negotiation.cc:240] Negotiation successful
0808 04:09:43.323516 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":336,"threads_started":1}
I20260808 04:09:43.326318 6472 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.83.61" port: 43419 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.5.83.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.60@KRBTEST.COM" created.
Aug 08 04:09:43 dist-test-slave-1l3l krb5kdc[6260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162183, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.61@KRBTEST.COM for kudu/127.5.83.61@KRBTEST.COM
I20260808 04:09:43.339208 6474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:43.326924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.61:43419 (local address 127.0.0.1:53324)
0808 04:09:43.327052 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0808 04:09:43.327066 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:09:43.327138 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:09:43.328050 (+ 912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:09:43.328054 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:09:43.328423 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:09:43.328544 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:09:43.328551 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:09:43.329833 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:09:43.329838 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:09:43.330652 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:09:43.330662 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:09:43.331390 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:09:43.331396 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:09:43.331584 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:43.332131 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:09:43.332151 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:09:43.334003 (+ 1852us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:09:43.336832 (+ 2829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:09:43.336837 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:09:43.336841 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:09:43.337264 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:09:43.337632 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:09:43.337637 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:09:43.337641 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:09:43.337778 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:09:43.338317 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:09:43.338322 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:09:43.338480 (+ 158us) client_negotiation.cc:769] Sending connection context
0808 04:09:43.338592 (+ 112us) client_negotiation.cc:240] Negotiation successful
0808 04:09:43.338839 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":2}
I20260808 04:09:43.339326 6476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:43.327173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53324 (local address 127.5.83.61:43419)
0808 04:09:43.327675 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:43.327681 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:43.327723 (+ 42us) server_negotiation.cc:407] Connection header received
0808 04:09:43.327769 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:43.327774 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:43.327919 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:43.328047 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:43.328712 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:43.329692 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:43.330773 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:43.331265 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:43.331370 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:43.334182 (+ 2812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:09:43.334192 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:09:43.334202 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:09:43.334229 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:09:43.336722 (+ 2493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:09:43.337408 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:09:43.337414 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:09:43.337417 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:09:43.337527 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:09:43.337875 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:09:43.337880 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:09:43.337883 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:09:43.338199 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:09:43.338390 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:43.338729 (+ 339us) server_negotiation.cc:299] Negotiation successful
0808 04:09:43.338955 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":371,"thread_start_us":258,"threads_started":1}
I20260808 04:09:43.341617 6472 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.83.60" port: 40519 } has no permanent_uuid. Determining permanent_uuid...
I20260808 04:09:43.342650 6474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:43.342243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.60:40519 (local address 127.0.0.1:51028)
0808 04:09:43.342391 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0808 04:09:43.342504 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.60:40519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260808 04:09:43.342965 6399 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.83.60:40519: connect: Connection refused (error 111)
I20260808 04:09:43.344439 6474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:43.344150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.60:40519 (local address 127.0.0.1:51032)
0808 04:09:43.344271 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0808 04:09:43.344343 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.60:40519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260808 04:09:43.344635 6472 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.83.60:40519: Network error: Client connection negotiation failed: client connection to 127.5.83.60:40519: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.60.keytab.
Entry for principal kudu/127.5.83.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.60.keytab.
I20260808 04:09:43.364737 5452 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2cd2tT/build/asan/bin/kudu
/tmp/dist-test-task2cd2tT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161826605655-5452-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161826605655-5452-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161826605655-5452-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161826605655-5452-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.60:40519
--webserver_interface=127.5.83.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.60.keytab
--principal=kudu/127.5.83.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.20:41623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.62:40575,127.5.83.61:43419,127.5.83.60:40519
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161826605655-5452-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161826605655-5452-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161826605655-5452-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:09:43.393914 6472 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.83.60" port: 40519 } attempt: 1
I20260808 04:09:43.395361 6474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:43.394725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.60:40519 (local address 127.0.0.1:51038)
0808 04:09:43.395049 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0808 04:09:43.395158 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.60:40519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153}
I20260808 04:09:43.396565 6474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:43.396235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.60:40519 (local address 127.0.0.1:51054)
0808 04:09:43.396378 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0808 04:09:43.396468 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.60:40519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260808 04:09:43.396759 6472 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.83.60:40519: Network error: Client connection negotiation failed: client connection to 127.5.83.60:40519: connect: Connection refused (error 111)
I20260808 04:09:43.464960 6472 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.83.60" port: 40519 } attempt: 2
I20260808 04:09:43.466025 6474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:43.465580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.60:40519 (local address 127.0.0.1:51066)
0808 04:09:43.465767 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0808 04:09:43.465866 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.60:40519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260808 04:09:43.467092 6474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:43.466766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.60:40519 (local address 127.0.0.1:51080)
0808 04:09:43.466905 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0808 04:09:43.466977 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.60:40519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260808 04:09:43.467278 6472 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.83.60:40519: Network error: Client connection negotiation failed: client connection to 127.5.83.60:40519: connect: Connection refused (error 111)
I20260808 04:09:43.502905 6367 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.83.61" port: 43419 } attempt: 5
Aug 08 04:09:43 dist-test-slave-1l3l krb5kdc[6260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162182, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.62@KRBTEST.COM for kudu/127.5.83.61@KRBTEST.COM
I20260808 04:09:43.515697 6476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:43.503831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53328 (local address 127.5.83.61:43419)
0808 04:09:43.504050 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:43.504054 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:43.504070 (+ 16us) server_negotiation.cc:407] Connection header received
0808 04:09:43.504120 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:43.504124 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:43.504181 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:43.504264 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:43.505076 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:43.505787 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:43.507007 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:43.507452 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:43.507563 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:43.510595 (+ 3032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:09:43.510600 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:09:43.510603 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:09:43.510634 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:09:43.513305 (+ 2671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:09:43.513911 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:09:43.513915 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:09:43.513917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:09:43.514034 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:09:43.514381 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:09:43.514386 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:09:43.514389 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:09:43.514651 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:09:43.514789 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:43.515172 (+ 383us) server_negotiation.cc:299] Negotiation successful
0808 04:09:43.515385 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260808 04:09:43.515697 6369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:43.503546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.61:43419 (local address 127.0.0.1:53328)
0808 04:09:43.503755 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0808 04:09:43.503773 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 04:09:43.503875 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:09:43.504340 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:09:43.504345 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:09:43.504734 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:09:43.504925 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:09:43.504935 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:09:43.505941 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:09:43.505946 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:09:43.506855 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:09:43.506869 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:09:43.507599 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:09:43.507605 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:09:43.507832 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:43.508489 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:09:43.508506 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:09:43.510424 (+ 1918us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:09:43.513432 (+ 3008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:09:43.513437 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:09:43.513442 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:09:43.513796 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:09:43.514137 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:09:43.514142 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:09:43.514146 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:09:43.514282 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:09:43.514763 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:09:43.514768 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:09:43.514919 (+ 151us) client_negotiation.cc:769] Sending connection context
0808 04:09:43.515057 (+ 138us) client_negotiation.cc:240] Negotiation successful
0808 04:09:43.515347 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260808 04:09:43.516844 6367 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.83.60" port: 40519 } has no permanent_uuid. Determining permanent_uuid...
I20260808 04:09:43.517855 6369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:43.517492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.60:40519 (local address 127.0.0.1:51082)
0808 04:09:43.517645 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0808 04:09:43.517723 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.60:40519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260808 04:09:43.518754 6369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:43.518453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.60:40519 (local address 127.0.0.1:51084)
0808 04:09:43.518587 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0808 04:09:43.518662 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.60:40519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260808 04:09:43.518949 6367 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.83.60:40519: Network error: Client connection negotiation failed: client connection to 127.5.83.60:40519: connect: Connection refused (error 111)
I20260808 04:09:43.558476 6472 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.83.60" port: 40519 } attempt: 3
I20260808 04:09:43.559666 6474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:43.559209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.60:40519 (local address 127.0.0.1:51096)
0808 04:09:43.559404 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0808 04:09:43.559498 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.60:40519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260808 04:09:43.560760 6474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:43.560306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.60:40519 (local address 127.0.0.1:51106)
0808 04:09:43.560544 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0808 04:09:43.560626 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.60:40519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
W20260808 04:09:43.560986 6472 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.83.60:40519: Network error: Client connection negotiation failed: client connection to 127.5.83.60:40519: connect: Connection refused (error 111)
I20260808 04:09:43.570150 6367 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.83.60" port: 40519 } attempt: 1
I20260808 04:09:43.571065 6369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:43.570663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.60:40519 (local address 127.0.0.1:51114)
0808 04:09:43.570826 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0808 04:09:43.570906 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.60:40519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260808 04:09:43.572002 6369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:43.571691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.60:40519 (local address 127.0.0.1:51124)
0808 04:09:43.571844 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0808 04:09:43.571912 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.60:40519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260808 04:09:43.572186 6367 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.83.60:40519: Network error: Client connection negotiation failed: client connection to 127.5.83.60:40519: connect: Connection refused (error 111)
I20260808 04:09:43.640399 6367 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.83.60" port: 40519 } attempt: 2
I20260808 04:09:43.641491 6369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:43.641025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.60:40519 (local address 127.0.0.1:51138)
0808 04:09:43.641238 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0808 04:09:43.641324 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.60:40519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260808 04:09:43.642721 6369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:43.642247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.60:40519 (local address 127.0.0.1:51154)
0808 04:09:43.642508 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0808 04:09:43.642592 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.60:40519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
W20260808 04:09:43.642970 6367 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.83.60:40519: Network error: Client connection negotiation failed: client connection to 127.5.83.60:40519: connect: Connection refused (error 111)
W20260808 04:09:43.694085 6479 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 04:09:43.694439 6479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:09:43.694542 6479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:09:43.694658 6479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:09:43.701786 6479 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:09:43.701879 6479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:09:43.701963 6479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:09:43.702028 6479 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:09:43.702116 6479 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:09:43.704205 6472 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.83.60" port: 40519 } attempt: 4
I20260808 04:09:43.705260 6474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:43.704799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.60:40519 (local address 127.0.0.1:51166)
0808 04:09:43.705024 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0808 04:09:43.705106 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.60:40519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260808 04:09:43.706130 6474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:43.705826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.60:40519 (local address 127.0.0.1:51180)
0808 04:09:43.705965 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0808 04:09:43.706029 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.60:40519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260808 04:09:43.706308 6472 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.83.60:40519: Network error: Client connection negotiation failed: client connection to 127.5.83.60:40519: connect: Connection refused (error 111)
I20260808 04:09:43.711807 6479 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.20:41623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161826605655-5452-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161826605655-5452-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.62:40575,127.5.83.61:43419,127.5.83.60:40519
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161826605655-5452-0/krb5kdc/kudu_127.5.83.60.keytab
--principal=kudu/127.5.83.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.60:40519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161826605655-5452-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.5.83.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161826605655-5452-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:09:43.713014 6479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:09:43.714280 6479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:09:43.725255 6486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:09:43.725292 6489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:09:43.725225 6487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:09:43.726032 6479 server_base.cc:1034] running on GCE node
I20260808 04:09:43.726748 6479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:09:43.728062 6479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:09:43.729272 6479 hybrid_clock.cc:648] HybridClock initialized: now 1786162183729212 us; error 38 us; skew 500 ppm
I20260808 04:09:43.729574 6479 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:09:43 dist-test-slave-1l3l krb5kdc[6260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162183, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:09:43.732498 6479 init.cc:377] Logged in from keytab as kudu/127.5.83.60@KRBTEST.COM (short username kudu)
I20260808 04:09:43.733855 6479 webserver.cc:460] Webserver started at http://127.5.83.60:41453/ using document root <none> and password file <none>
I20260808 04:09:43.734468 6479 fs_manager.cc:359] Metadata directory not provided
I20260808 04:09:43.734567 6479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:09:43.734861 6479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:09:43.737725 6479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161826605655-5452-0/minicluster-data/master-2/data/instance:
uuid: "cb7eff48da4746c889ee7e9e791e3afa"
format_stamp: "Formatted at 2026-08-08 04:09:43 on dist-test-slave-1l3l"
server_key: "70d15ef70c51e4febceacdcd95b1a778"
server_key_iv: "8fdf520d22a2d6c7b53943317622078f"
server_key_version: "encryptionkey@0"
I20260808 04:09:43.738458 6479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161826605655-5452-0/minicluster-data/master-2/wal/instance:
uuid: "cb7eff48da4746c889ee7e9e791e3afa"
format_stamp: "Formatted at 2026-08-08 04:09:43 on dist-test-slave-1l3l"
server_key: "70d15ef70c51e4febceacdcd95b1a778"
server_key_iv: "8fdf520d22a2d6c7b53943317622078f"
server_key_version: "encryptionkey@0"
I20260808 04:09:43.743142 6479 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260808 04:09:43.746335 6497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:09:43.747522 6479 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260808 04:09:43.747670 6479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161826605655-5452-0/minicluster-data/master-2/data,/tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161826605655-5452-0/minicluster-data/master-2/wal
uuid: "cb7eff48da4746c889ee7e9e791e3afa"
format_stamp: "Formatted at 2026-08-08 04:09:43 on dist-test-slave-1l3l"
server_key: "70d15ef70c51e4febceacdcd95b1a778"
server_key_iv: "8fdf520d22a2d6c7b53943317622078f"
server_key_version: "encryptionkey@0"
I20260808 04:09:43.747893 6479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161826605655-5452-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161826605655-5452-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161826605655-5452-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:09:43.749181 6367 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.83.60" port: 40519 } attempt: 3
I20260808 04:09:43.750187 6369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:43.749753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.60:40519 (local address 127.0.0.1:51194)
0808 04:09:43.749946 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0808 04:09:43.750034 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.60:40519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260808 04:09:43.751145 6369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:43.750784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.60:40519 (local address 127.0.0.1:51208)
0808 04:09:43.750976 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0808 04:09:43.751046 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.60:40519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260808 04:09:43.751348 6367 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.83.60:40519: Network error: Client connection negotiation failed: client connection to 127.5.83.60:40519: connect: Connection refused (error 111)
I20260808 04:09:43.762296 6479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:09:43.766111 6479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:09:43.766335 6479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:09:43.782963 6479 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.60:40519
I20260808 04:09:43.782980 6566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.60:40519 every 8 connection(s)
I20260808 04:09:43.784255 6479 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161826605655-5452-0/minicluster-data/master-2/data/info.pb
I20260808 04:09:43.787567 6567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:09:43.790220 5452 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2cd2tT/build/asan/bin/kudu as pid 6479
I20260808 04:09:43.790409 5452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2cd2tT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161826605655-5452-0/minicluster-data/master-2/wal/instance
I20260808 04:09:43.791031 5452 external_mini_cluster.cc:1428] Setting key 5afb74dd267bced496c0e7e7bf9b8d52
I20260808 04:09:43.794874 6567 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.83.62" port: 40575 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 04:09:43 dist-test-slave-1l3l krb5kdc[6260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162183, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.60@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
Aug 08 04:09:43 dist-test-slave-1l3l krb5kdc[6260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.60@KRBTEST.COM
I20260808 04:09:43.813722 6370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:43.797722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40528 (local address 127.5.83.62:40575)
0808 04:09:43.797965 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:43.797971 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:43.798629 (+ 658us) server_negotiation.cc:407] Connection header received
0808 04:09:43.798815 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:43.798820 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:43.798884 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:43.798970 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:43.800197 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:43.801146 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:43.802847 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:43.803501 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:43.803609 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:43.807263 (+ 3654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:09:43.807269 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:09:43.807272 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:09:43.807303 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:09:43.810647 (+ 3344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:09:43.811549 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:09:43.811555 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:09:43.811558 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:09:43.811748 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:09:43.812141 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:09:43.812146 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:09:43.812150 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:09:43.812409 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:09:43.812537 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:43.813160 (+ 623us) server_negotiation.cc:299] Negotiation successful
0808 04:09:43.813372 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 04:09:43.814640 6572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:43.792729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51216 (local address 127.5.83.60:40519)
0808 04:09:43.793440 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:43.793450 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:43.793488 (+ 38us) server_negotiation.cc:407] Connection header received
0808 04:09:43.795074 (+ 1586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:43.795095 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:43.795493 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:43.795851 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:43.797538 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:43.799414 (+ 1876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:43.801030 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:43.801951 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:43.802120 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:43.808157 (+ 6037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:09:43.808167 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:09:43.808182 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:09:43.808218 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:09:43.811513 (+ 3295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:09:43.812184 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:09:43.812190 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:09:43.812192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:09:43.812311 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:09:43.812666 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:09:43.812669 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:09:43.812671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:09:43.813005 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:09:43.813290 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:43.813584 (+ 294us) server_negotiation.cc:299] Negotiation successful
0808 04:09:43.813815 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":323,"threads_started":1}
I20260808 04:09:43.814657 6573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:43.797744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.62:40575 (local address 127.0.0.1:40528)
0808 04:09:43.798486 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0808 04:09:43.798534 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 04:09:43.798676 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:09:43.799048 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:09:43.799055 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:09:43.799803 (+ 748us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:09:43.800055 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:09:43.800064 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:09:43.801280 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:09:43.801286 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:09:43.802701 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:09:43.802716 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:09:43.803699 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:09:43.803730 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:09:43.803917 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:43.804552 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:09:43.804581 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:09:43.807078 (+ 2497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:09:43.810779 (+ 3701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:09:43.810792 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:09:43.810799 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:09:43.811419 (+ 620us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:09:43.811851 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:09:43.811856 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:09:43.811860 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:09:43.812035 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:09:43.812532 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:09:43.812539 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:09:43.812817 (+ 278us) client_negotiation.cc:769] Sending connection context
0808 04:09:43.813047 (+ 230us) client_negotiation.cc:240] Negotiation successful
0808 04:09:43.813450 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":343,"threads_started":1}
I20260808 04:09:43.816095 6567 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.83.61" port: 43419 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 04:09:43 dist-test-slave-1l3l krb5kdc[6260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162183, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.60@KRBTEST.COM for kudu/127.5.83.61@KRBTEST.COM
Aug 08 04:09:43 dist-test-slave-1l3l krb5kdc[6260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.62@KRBTEST.COM
I20260808 04:09:43.828552 6573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:43.816621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.61:43419 (local address 127.0.0.1:53338)
0808 04:09:43.816828 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0808 04:09:43.816844 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:09:43.816929 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:09:43.817224 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:09:43.817228 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:09:43.817636 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:09:43.817803 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:09:43.817814 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:09:43.818813 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:09:43.818818 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:09:43.819729 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:09:43.819744 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:09:43.820480 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:09:43.820486 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:09:43.820660 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:43.821122 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:09:43.821143 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:09:43.823114 (+ 1971us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:09:43.826392 (+ 3278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:09:43.826397 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:09:43.826400 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:09:43.826724 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:09:43.827081 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:09:43.827086 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:09:43.827089 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:09:43.827222 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:09:43.827687 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:09:43.827691 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:09:43.827843 (+ 152us) client_negotiation.cc:769] Sending connection context
0808 04:09:43.827965 (+ 122us) client_negotiation.cc:240] Negotiation successful
0808 04:09:43.828209 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119}
I20260808 04:09:43.828642 6476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:43.816748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53338 (local address 127.5.83.61:43419)
0808 04:09:43.816929 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:43.816935 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:43.816956 (+ 21us) server_negotiation.cc:407] Connection header received
0808 04:09:43.817017 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:43.817023 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:43.817089 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:43.817191 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:43.818006 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:43.818661 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:43.819847 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:43.820329 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:43.820469 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:43.823281 (+ 2812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:09:43.823286 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:09:43.823289 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:09:43.823317 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:09:43.826281 (+ 2964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:09:43.826830 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:09:43.826834 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:09:43.826838 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:09:43.826977 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:09:43.827321 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:09:43.827324 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:09:43.827326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:09:43.827577 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:09:43.827733 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:43.828075 (+ 342us) server_negotiation.cc:299] Negotiation successful
0808 04:09:43.828280 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 04:09:43.829571 6567 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.83.60" port: 40519 } has no permanent_uuid. Determining permanent_uuid...
I20260808 04:09:43.830864 6370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:43.817292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40542 (local address 127.5.83.62:40575)
0808 04:09:43.817435 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:43.817438 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:43.817568 (+ 130us) server_negotiation.cc:407] Connection header received
0808 04:09:43.817703 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:43.817706 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:43.817753 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:43.817820 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:43.818930 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:43.819622 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:43.821154 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:43.821538 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:43.821625 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:43.825602 (+ 3977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:09:43.825608 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:09:43.825612 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:09:43.825655 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:09:43.828231 (+ 2576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:09:43.828883 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:09:43.828887 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:09:43.828889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:09:43.829012 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:09:43.829355 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:09:43.829360 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:09:43.829363 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:09:43.829654 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:09:43.829881 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:43.830376 (+ 495us) server_negotiation.cc:299] Negotiation successful
0808 04:09:43.830581 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
Aug 08 04:09:43 dist-test-slave-1l3l krb5kdc[6260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162183, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.60@KRBTEST.COM for kudu/127.5.83.60@KRBTEST.COM
Aug 08 04:09:43 dist-test-slave-1l3l krb5kdc[6260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.61@KRBTEST.COM
I20260808 04:09:43.842149 6573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:43.830292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.60:40519 (local address 127.0.0.1:51226)
0808 04:09:43.830448 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0808 04:09:43.830466 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 04:09:43.830554 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:09:43.830906 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:09:43.830910 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:09:43.831266 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:09:43.831438 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:09:43.831450 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:09:43.832475 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:09:43.832480 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:09:43.833292 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:09:43.833303 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:09:43.833939 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:09:43.833945 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:09:43.834131 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:43.834600 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:09:43.834615 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:09:43.836605 (+ 1990us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:09:43.839821 (+ 3216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:09:43.839826 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:09:43.839831 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:09:43.840299 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:09:43.840637 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:09:43.840642 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:09:43.840645 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:09:43.840799 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:09:43.841249 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:09:43.841253 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:09:43.841368 (+ 115us) client_negotiation.cc:769] Sending connection context
0808 04:09:43.841498 (+ 130us) client_negotiation.cc:240] Negotiation successful
0808 04:09:43.841745 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260808 04:09:43.842209 6572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:43.830524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51226 (local address 127.5.83.60:40519)
0808 04:09:43.830661 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:43.830666 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:43.830682 (+ 16us) server_negotiation.cc:407] Connection header received
0808 04:09:43.830725 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:43.830729 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:43.830789 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:43.830870 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:43.831587 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:43.832323 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:43.833414 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:43.833813 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:43.833926 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:43.836782 (+ 2856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:09:43.836788 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:09:43.836793 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:09:43.836827 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:09:43.839658 (+ 2831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:09:43.840410 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:09:43.840413 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:09:43.840415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:09:43.840519 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:09:43.840903 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:09:43.840906 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:09:43.840908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:09:43.841142 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:09:43.841267 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:43.841590 (+ 323us) server_negotiation.cc:299] Negotiation successful
0808 04:09:43.841913 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":104,"server-negotiator.queue_time_us":51}
I20260808 04:09:43.846269 6476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:43.832459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53352 (local address 127.5.83.61:43419)
0808 04:09:43.832625 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:43.832630 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:43.832646 (+ 16us) server_negotiation.cc:407] Connection header received
0808 04:09:43.832687 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:43.832691 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:43.832737 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:43.832812 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:43.833926 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:43.834622 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:43.835758 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:43.836284 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:43.836421 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:43.841146 (+ 4725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:09:43.841151 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:09:43.841156 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:09:43.841183 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:09:43.843982 (+ 2799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:09:43.844579 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:09:43.844584 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:09:43.844587 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:09:43.844723 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:09:43.845021 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:09:43.845026 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:09:43.845029 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:09:43.845325 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:09:43.845570 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:43.845743 (+ 173us) server_negotiation.cc:299] Negotiation successful
0808 04:09:43.845948 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 04:09:43.848049 6567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb7eff48da4746c889ee7e9e791e3afa: Bootstrap starting.
I20260808 04:09:43.850728 5452 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:09:43.851626 6567 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb7eff48da4746c889ee7e9e791e3afa: Neither blocks nor log segments found. Creating new log.
I20260808 04:09:43.852768 6567 log.cc:824] T 00000000000000000000000000000000 P cb7eff48da4746c889ee7e9e791e3afa: Log is configured to *not* fsync() on all Append() calls
I20260808 04:09:43.855877 6567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb7eff48da4746c889ee7e9e791e3afa: No bootstrap required, opened a new log
I20260808 04:09:43.853192 6330 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "9dcb00ee5483403f89c52903124ef8ba"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40542
I20260808 04:09:43.866263 6567 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb7eff48da4746c889ee7e9e791e3afa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcb00ee5483403f89c52903124ef8ba" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 40575 } } peers { permanent_uuid: "6facd467610c48b4b5f053749ecd7b59" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 43419 } } peers { permanent_uuid: "cb7eff48da4746c889ee7e9e791e3afa" member_type: VOTER last_known_addr { host: "127.5.83.60" port: 40519 } }
I20260808 04:09:43.866644 6567 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb7eff48da4746c889ee7e9e791e3afa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb7eff48da4746c889ee7e9e791e3afa, State: Initialized, Role: FOLLOWER
I20260808 04:09:43.867106 6567 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb7eff48da4746c889ee7e9e791e3afa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcb00ee5483403f89c52903124ef8ba" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 40575 } } peers { permanent_uuid: "6facd467610c48b4b5f053749ecd7b59" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 43419 } } peers { permanent_uuid: "cb7eff48da4746c889ee7e9e791e3afa" member_type: VOTER last_known_addr { host: "127.5.83.60" port: 40519 } }
I20260808 04:09:43.867992 6577 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb7eff48da4746c889ee7e9e791e3afa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcb00ee5483403f89c52903124ef8ba" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 40575 } } peers { permanent_uuid: "6facd467610c48b4b5f053749ecd7b59" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 43419 } } peers { permanent_uuid: "cb7eff48da4746c889ee7e9e791e3afa" member_type: VOTER last_known_addr { host: "127.5.83.60" port: 40519 } } }
I20260808 04:09:43.868294 6577 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb7eff48da4746c889ee7e9e791e3afa [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:09:43.868990 6567 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb7eff48da4746c889ee7e9e791e3afa [sys.catalog]: configured and running, proceeding with master startup.
W20260808 04:09:43.877482 6591 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cb7eff48da4746c889ee7e9e791e3afa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 04:09:43.877602 6591 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 04:09:43.875555 6479 master_runner.cc:186] Error getting master registration for 127.5.83.62:40575: OK, instance_id { permanent_uuid: "9dcb00ee5483403f89c52903124ef8ba" instance_seqno: 1786162182797869 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 04:09:43.886090 6479 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 04:09:43.928587 6367 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.83.60" port: 40519 } attempt: 4
Aug 08 04:09:43 dist-test-slave-1l3l krb5kdc[6260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162182, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.62@KRBTEST.COM for kudu/127.5.83.60@KRBTEST.COM
I20260808 04:09:43.945250 6369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:43.929311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.60:40519 (local address 127.0.0.1:51228)
0808 04:09:43.929644 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0808 04:09:43.929661 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:09:43.929764 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:09:43.930163 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:09:43.930168 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:09:43.930684 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:09:43.930911 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:09:43.930924 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:09:43.932380 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:09:43.932386 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:09:43.933873 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:09:43.933897 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:09:43.934864 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:09:43.934869 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:09:43.935084 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:43.935810 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:09:43.935832 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:09:43.938676 (+ 2844us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:09:43.942561 (+ 3885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:09:43.942566 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:09:43.942571 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:09:43.943053 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:09:43.943422 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:09:43.943427 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:09:43.943431 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:09:43.943594 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:09:43.944295 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:09:43.944300 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:09:43.944461 (+ 161us) client_negotiation.cc:769] Sending connection context
0808 04:09:43.944600 (+ 139us) client_negotiation.cc:240] Negotiation successful
0808 04:09:43.944869 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216}
I20260808 04:09:43.945299 6572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:43.929447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51228 (local address 127.5.83.60:40519)
0808 04:09:43.929677 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:43.929683 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:43.929710 (+ 27us) server_negotiation.cc:407] Connection header received
0808 04:09:43.929881 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:43.929887 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:43.929971 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:43.930094 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:43.931067 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:43.932246 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:43.934027 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:43.934730 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:43.934859 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:43.938862 (+ 4003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:09:43.938869 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:09:43.938874 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:09:43.938913 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:09:43.942428 (+ 3515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:09:43.943165 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:09:43.943171 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:09:43.943175 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:09:43.943319 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:09:43.943698 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:09:43.943732 (+ 34us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:09:43.943736 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:09:43.944164 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:09:43.944350 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:43.944725 (+ 375us) server_negotiation.cc:299] Negotiation successful
0808 04:09:43.944946 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 04:09:43.950105 6367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dcb00ee5483403f89c52903124ef8ba: Bootstrap starting.
I20260808 04:09:43.953562 6367 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9dcb00ee5483403f89c52903124ef8ba: Neither blocks nor log segments found. Creating new log.
I20260808 04:09:43.954560 6367 log.cc:824] T 00000000000000000000000000000000 P 9dcb00ee5483403f89c52903124ef8ba: Log is configured to *not* fsync() on all Append() calls
I20260808 04:09:43.957113 6367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dcb00ee5483403f89c52903124ef8ba: No bootstrap required, opened a new log
I20260808 04:09:43.961905 6367 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9dcb00ee5483403f89c52903124ef8ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcb00ee5483403f89c52903124ef8ba" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 40575 } } peers { permanent_uuid: "6facd467610c48b4b5f053749ecd7b59" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 43419 } } peers { permanent_uuid: "cb7eff48da4746c889ee7e9e791e3afa" member_type: VOTER last_known_addr { host: "127.5.83.60" port: 40519 } }
I20260808 04:09:43.962185 6367 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9dcb00ee5483403f89c52903124ef8ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dcb00ee5483403f89c52903124ef8ba, State: Initialized, Role: FOLLOWER
I20260808 04:09:43.962570 6367 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9dcb00ee5483403f89c52903124ef8ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcb00ee5483403f89c52903124ef8ba" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 40575 } } peers { permanent_uuid: "6facd467610c48b4b5f053749ecd7b59" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 43419 } } peers { permanent_uuid: "cb7eff48da4746c889ee7e9e791e3afa" member_type: VOTER last_known_addr { host: "127.5.83.60" port: 40519 } }
I20260808 04:09:43.963392 6596 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dcb00ee5483403f89c52903124ef8ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcb00ee5483403f89c52903124ef8ba" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 40575 } } peers { permanent_uuid: "6facd467610c48b4b5f053749ecd7b59" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 43419 } } peers { permanent_uuid: "cb7eff48da4746c889ee7e9e791e3afa" member_type: VOTER last_known_addr { host: "127.5.83.60" port: 40519 } } }
I20260808 04:09:43.963794 6596 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dcb00ee5483403f89c52903124ef8ba [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:09:43.964444 6367 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9dcb00ee5483403f89c52903124ef8ba [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:09:43.964555 6330 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9dcb00ee5483403f89c52903124ef8ba [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 04:09:43.964756 6330 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9dcb00ee5483403f89c52903124ef8ba [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:09:43.968003 6330 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9dcb00ee5483403f89c52903124ef8ba [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcb00ee5483403f89c52903124ef8ba" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 40575 } } peers { permanent_uuid: "6facd467610c48b4b5f053749ecd7b59" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 43419 } } peers { permanent_uuid: "cb7eff48da4746c889ee7e9e791e3afa" member_type: VOTER last_known_addr { host: "127.5.83.60" port: 40519 } }
I20260808 04:09:43.970422 6536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9dcb00ee5483403f89c52903124ef8ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "cb7eff48da4746c889ee7e9e791e3afa"
I20260808 04:09:43.970891 6536 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb7eff48da4746c889ee7e9e791e3afa [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:09:43.974269 6536 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cb7eff48da4746c889ee7e9e791e3afa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dcb00ee5483403f89c52903124ef8ba in term 1.
I20260808 04:09:43.974903 6299 leader_election.cc:302] T 00000000000000000000000000000000 P 9dcb00ee5483403f89c52903124ef8ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dcb00ee5483403f89c52903124ef8ba, cb7eff48da4746c889ee7e9e791e3afa; no voters:
I20260808 04:09:43.975356 6596 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9dcb00ee5483403f89c52903124ef8ba [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:09:43.975383 6330 leader_election.cc:288] T 00000000000000000000000000000000 P 9dcb00ee5483403f89c52903124ef8ba [CANDIDATE]: Term 1 election: Requested vote from peers 6facd467610c48b4b5f053749ecd7b59 (127.5.83.61:43419), cb7eff48da4746c889ee7e9e791e3afa (127.5.83.60:40519)
I20260808 04:09:44.003044 6596 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9dcb00ee5483403f89c52903124ef8ba [term 1 LEADER]: Becoming Leader. State: Replica: 9dcb00ee5483403f89c52903124ef8ba, State: Running, Role: LEADER
I20260808 04:09:44.005487 6472 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.83.60" port: 40519 } attempt: 5
I20260808 04:09:44.007185 6596 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9dcb00ee5483403f89c52903124ef8ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcb00ee5483403f89c52903124ef8ba" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 40575 } } peers { permanent_uuid: "6facd467610c48b4b5f053749ecd7b59" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 43419 } } peers { permanent_uuid: "cb7eff48da4746c889ee7e9e791e3afa" member_type: VOTER last_known_addr { host: "127.5.83.60" port: 40519 } }
W20260808 04:09:44.018182 6503 tablet.cc:2369] T 00000000000000000000000000000000 P cb7eff48da4746c889ee7e9e791e3afa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:09:44.020589 6604 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dcb00ee5483403f89c52903124ef8ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9dcb00ee5483403f89c52903124ef8ba. Latest consensus state: current_term: 1 leader_uuid: "9dcb00ee5483403f89c52903124ef8ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcb00ee5483403f89c52903124ef8ba" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 40575 } } peers { permanent_uuid: "6facd467610c48b4b5f053749ecd7b59" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 43419 } } peers { permanent_uuid: "cb7eff48da4746c889ee7e9e791e3afa" member_type: VOTER last_known_addr { host: "127.5.83.60" port: 40519 } } }
I20260808 04:09:44.021060 6604 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dcb00ee5483403f89c52903124ef8ba [sys.catalog]: This master's current role is: LEADER
I20260808 04:09:44.024765 6572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:43.993603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51230 (local address 127.5.83.60:40519)
0808 04:09:43.993781 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:43.993787 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:44.000253 (+ 6466us) server_negotiation.cc:407] Connection header received
0808 04:09:44.000544 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:44.000550 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:44.000637 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:44.000715 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:44.003878 (+ 3163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:44.005013 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:44.009323 (+ 4310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:44.012620 (+ 3297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:44.012734 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:44.016101 (+ 3367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:09:44.016107 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:09:44.016112 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:09:44.016147 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:09:44.019756 (+ 3609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:09:44.020736 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:09:44.020741 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:09:44.020745 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:09:44.020903 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:09:44.021720 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:09:44.021725 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:09:44.021729 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:09:44.022066 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:09:44.022302 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:44.024164 (+ 1862us) server_negotiation.cc:299] Negotiation successful
0808 04:09:44.024382 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 04:09:44.031337 6624 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Aug 08 04:09:44 dist-test-slave-1l3l krb5kdc[6260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786162183, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.61@KRBTEST.COM for kudu/127.5.83.60@KRBTEST.COM
I20260808 04:09:44.035058 6476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:44.000803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53360 (local address 127.5.83.61:43419)
0808 04:09:44.000980 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:44.000986 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:44.006557 (+ 5571us) server_negotiation.cc:407] Connection header received
0808 04:09:44.006891 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:44.006896 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:44.006970 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:44.007047 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:44.009665 (+ 2618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:44.010669 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:44.012428 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:44.013025 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:44.013117 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:44.027851 (+ 14734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:09:44.027857 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:09:44.027862 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:09:44.027906 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:09:44.031477 (+ 3571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:09:44.032327 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:09:44.032332 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:09:44.032336 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:09:44.032488 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:09:44.032905 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:09:44.032910 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:09:44.032913 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:09:44.033268 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:09:44.033518 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:44.034492 (+ 974us) server_negotiation.cc:299] Negotiation successful
0808 04:09:44.034733 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260808 04:09:44.038141 6306 tablet.cc:2369] T 00000000000000000000000000000000 P 9dcb00ee5483403f89c52903124ef8ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:09:44.044776 6370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:43.995061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40556 (local address 127.5.83.62:40575)
0808 04:09:43.995244 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:43.995251 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:44.002586 (+ 7335us) server_negotiation.cc:407] Connection header received
0808 04:09:44.003683 (+ 1097us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:44.003689 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:44.003785 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:44.003856 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:44.009083 (+ 5227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:44.010167 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:44.016250 (+ 6083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:44.016953 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:44.017052 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:44.034835 (+ 17783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:09:44.034841 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:09:44.034846 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:09:44.034892 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:09:44.038568 (+ 3676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:09:44.040340 (+ 1772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:09:44.040346 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:09:44.040349 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:09:44.040557 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:09:44.041651 (+ 1094us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:09:44.041660 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:09:44.041664 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:09:44.042391 (+ 727us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:09:44.042801 (+ 410us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:44.043737 (+ 936us) server_negotiation.cc:299] Negotiation successful
0808 04:09:44.044136 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 04:09:44.053581 6474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:44.006025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.60:40519 (local address 127.0.0.1:51242)
0808 04:09:44.006357 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0808 04:09:44.006373 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:09:44.006476 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:09:44.018183 (+ 11707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:09:44.018188 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:09:44.018671 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:09:44.018852 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:09:44.018864 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:09:44.027807 (+ 8943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:09:44.027812 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:09:44.029589 (+ 1777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:09:44.029604 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:09:44.031288 (+ 1684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:09:44.031294 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:09:44.031497 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:44.032075 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:09:44.032098 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:09:44.035524 (+ 3426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:09:44.043799 (+ 8275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:09:44.043803 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:09:44.043807 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:09:44.044318 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:09:44.051804 (+ 7486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:09:44.051809 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:09:44.051812 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:09:44.052009 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:09:44.052643 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:09:44.052648 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:09:44.052803 (+ 155us) client_negotiation.cc:769] Sending connection context
0808 04:09:44.052949 (+ 146us) client_negotiation.cc:240] Negotiation successful
0808 04:09:44.053219 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223}
I20260808 04:09:44.054636 6619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:44.006678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51242 (local address 127.5.83.60:40519)
0808 04:09:44.015916 (+ 9238us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:44.015922 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:44.015946 (+ 24us) server_negotiation.cc:407] Connection header received
0808 04:09:44.015992 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:44.015997 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:44.016068 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:44.019010 (+ 2942us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:44.019303 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:44.020513 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:44.030041 (+ 9528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:44.030649 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:44.038715 (+ 8066us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:44.038955 (+ 240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:09:44.038962 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:09:44.038967 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:09:44.038999 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:09:44.043316 (+ 4317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:09:44.044480 (+ 1164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:09:44.044485 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:09:44.044488 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:09:44.044627 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:09:44.052148 (+ 7521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:09:44.052152 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:09:44.052155 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:09:44.052504 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:09:44.053929 (+ 1425us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:44.054048 (+ 119us) server_negotiation.cc:299] Negotiation successful
0808 04:09:44.054255 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9098,"thread_start_us":289,"threads_started":1}
I20260808 04:09:44.058967 6624 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:09:44.062148 6472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6facd467610c48b4b5f053749ecd7b59: Bootstrap starting.
I20260808 04:09:44.070165 6472 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6facd467610c48b4b5f053749ecd7b59: Neither blocks nor log segments found. Creating new log.
I20260808 04:09:44.072249 6472 log.cc:824] T 00000000000000000000000000000000 P 6facd467610c48b4b5f053749ecd7b59: Log is configured to *not* fsync() on all Append() calls
I20260808 04:09:44.087663 6472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6facd467610c48b4b5f053749ecd7b59: No bootstrap required, opened a new log
I20260808 04:09:44.102792 6472 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6facd467610c48b4b5f053749ecd7b59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcb00ee5483403f89c52903124ef8ba" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 40575 } } peers { permanent_uuid: "6facd467610c48b4b5f053749ecd7b59" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 43419 } } peers { permanent_uuid: "cb7eff48da4746c889ee7e9e791e3afa" member_type: VOTER last_known_addr { host: "127.5.83.60" port: 40519 } }
I20260808 04:09:44.103188 6472 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6facd467610c48b4b5f053749ecd7b59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6facd467610c48b4b5f053749ecd7b59, State: Initialized, Role: FOLLOWER
I20260808 04:09:44.103695 6472 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6facd467610c48b4b5f053749ecd7b59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcb00ee5483403f89c52903124ef8ba" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 40575 } } peers { permanent_uuid: "6facd467610c48b4b5f053749ecd7b59" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 43419 } } peers { permanent_uuid: "cb7eff48da4746c889ee7e9e791e3afa" member_type: VOTER last_known_addr { host: "127.5.83.60" port: 40519 } }
I20260808 04:09:44.106736 6536 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cb7eff48da4746c889ee7e9e791e3afa [term 1 FOLLOWER]: Refusing update from remote peer 9dcb00ee5483403f89c52903124ef8ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:09:44.107779 6604 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9dcb00ee5483403f89c52903124ef8ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb7eff48da4746c889ee7e9e791e3afa" member_type: VOTER last_known_addr { host: "127.5.83.60" port: 40519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:09:44.110641 6633 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6facd467610c48b4b5f053749ecd7b59 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcb00ee5483403f89c52903124ef8ba" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 40575 } } peers { permanent_uuid: "6facd467610c48b4b5f053749ecd7b59" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 43419 } } peers { permanent_uuid: "cb7eff48da4746c889ee7e9e791e3afa" member_type: VOTER last_known_addr { host: "127.5.83.60" port: 40519 } } }
I20260808 04:09:44.113536 6472 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6facd467610c48b4b5f053749ecd7b59 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:09:44.113567 6433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9dcb00ee5483403f89c52903124ef8ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "6facd467610c48b4b5f053749ecd7b59"
I20260808 04:09:44.114051 6432 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6facd467610c48b4b5f053749ecd7b59 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:09:44.111019 6633 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6facd467610c48b4b5f053749ecd7b59 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:09:44.117120 6432 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6facd467610c48b4b5f053749ecd7b59 [term 1 FOLLOWER]: Refusing update from remote peer 9dcb00ee5483403f89c52903124ef8ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:09:44.120265 6433 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6facd467610c48b4b5f053749ecd7b59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dcb00ee5483403f89c52903124ef8ba in term 1.
I20260808 04:09:44.121248 6276 master_runner.cc:186] Error getting master registration for 127.5.83.61:43419: OK, instance_id { permanent_uuid: "6facd467610c48b4b5f053749ecd7b59" instance_seqno: 1786162183286036 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 04:09:44.127664 6276 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 04:09:44.127907 6604 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9dcb00ee5483403f89c52903124ef8ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "6facd467610c48b4b5f053749ecd7b59" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 43419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:09:44.138386 6577 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb7eff48da4746c889ee7e9e791e3afa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9dcb00ee5483403f89c52903124ef8ba. Latest consensus state: current_term: 1 leader_uuid: "9dcb00ee5483403f89c52903124ef8ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcb00ee5483403f89c52903124ef8ba" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 40575 } } peers { permanent_uuid: "6facd467610c48b4b5f053749ecd7b59" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 43419 } } peers { permanent_uuid: "cb7eff48da4746c889ee7e9e791e3afa" member_type: VOTER last_known_addr { host: "127.5.83.60" port: 40519 } } }
I20260808 04:09:44.138676 6577 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb7eff48da4746c889ee7e9e791e3afa [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:09:44.144295 6629 mvcc.cc:205] Tried to move back new op lower bound from 7316120306088742912 to 7316120305703972864. Current Snapshot: MvccSnapshot[applied={T|T < 7316120306088742912}]
I20260808 04:09:44.155388 6577 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb7eff48da4746c889ee7e9e791e3afa [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9dcb00ee5483403f89c52903124ef8ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcb00ee5483403f89c52903124ef8ba" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 40575 } } peers { permanent_uuid: "6facd467610c48b4b5f053749ecd7b59" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 43419 } } peers { permanent_uuid: "cb7eff48da4746c889ee7e9e791e3afa" member_type: VOTER last_known_addr { host: "127.5.83.60" port: 40519 } } }
I20260808 04:09:44.155646 6577 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb7eff48da4746c889ee7e9e791e3afa [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:09:44.166074 6604 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dcb00ee5483403f89c52903124ef8ba [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9dcb00ee5483403f89c52903124ef8ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcb00ee5483403f89c52903124ef8ba" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 40575 } } peers { permanent_uuid: "6facd467610c48b4b5f053749ecd7b59" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 43419 } } peers { permanent_uuid: "cb7eff48da4746c889ee7e9e791e3afa" member_type: VOTER last_known_addr { host: "127.5.83.60" port: 40519 } } }
I20260808 04:09:44.166354 6604 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dcb00ee5483403f89c52903124ef8ba [sys.catalog]: This master's current role is: LEADER
I20260808 04:09:44.168821 6633 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6facd467610c48b4b5f053749ecd7b59 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9dcb00ee5483403f89c52903124ef8ba. Latest consensus state: current_term: 1 leader_uuid: "9dcb00ee5483403f89c52903124ef8ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcb00ee5483403f89c52903124ef8ba" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 40575 } } peers { permanent_uuid: "6facd467610c48b4b5f053749ecd7b59" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 43419 } } peers { permanent_uuid: "cb7eff48da4746c889ee7e9e791e3afa" member_type: VOTER last_known_addr { host: "127.5.83.60" port: 40519 } } }
I20260808 04:09:44.169175 6633 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6facd467610c48b4b5f053749ecd7b59 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:09:44.170578 6624 catalog_manager.cc:1347] Generated new cluster ID: 0e4eea5ac4aa4d8e9762e0c1ec51022f
I20260808 04:09:44.170691 6624 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:09:44.174831 6646 mvcc.cc:205] Tried to move back new op lower bound from 7316120306088742912 to 7316120305703972864. Current Snapshot: MvccSnapshot[applied={T|T < 7316120306088742912}]
I20260808 04:09:44.175849 6633 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6facd467610c48b4b5f053749ecd7b59 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9dcb00ee5483403f89c52903124ef8ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcb00ee5483403f89c52903124ef8ba" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 40575 } } peers { permanent_uuid: "6facd467610c48b4b5f053749ecd7b59" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 43419 } } peers { permanent_uuid: "cb7eff48da4746c889ee7e9e791e3afa" member_type: VOTER last_known_addr { host: "127.5.83.60" port: 40519 } } }
I20260808 04:09:44.176201 6633 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6facd467610c48b4b5f053749ecd7b59 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:09:44.176746 6657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dcb00ee5483403f89c52903124ef8ba [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9dcb00ee5483403f89c52903124ef8ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcb00ee5483403f89c52903124ef8ba" member_type: VOTER last_known_addr { host: "127.5.83.62" port: 40575 } } peers { permanent_uuid: "6facd467610c48b4b5f053749ecd7b59" member_type: VOTER last_known_addr { host: "127.5.83.61" port: 43419 } } peers { permanent_uuid: "cb7eff48da4746c889ee7e9e791e3afa" member_type: VOTER last_known_addr { host: "127.5.83.60" port: 40519 } } }
I20260808 04:09:44.177495 6657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dcb00ee5483403f89c52903124ef8ba [sys.catalog]: This master's current role is: LEADER
I20260808 04:09:44.202557 6624 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:09:44.204634 6624 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:09:44.221333 6624 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9dcb00ee5483403f89c52903124ef8ba: Generated new TSK 0
I20260808 04:09:44.222340 6624 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:09:44.227161 6659 catalog_manager.cc:1259] Loaded cluster ID: 0e4eea5ac4aa4d8e9762e0c1ec51022f
I20260808 04:09:44.227257 6659 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6facd467610c48b4b5f053749ecd7b59: loading cluster ID for follower catalog manager: success
I20260808 04:09:44.231550 6659 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6facd467610c48b4b5f053749ecd7b59: acquiring CA information for follower catalog manager: success
I20260808 04:09:44.233855 6659 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6facd467610c48b4b5f053749ecd7b59: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260808 04:09:44.362627 6619 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:09:44.348786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51250 (local address 127.5.83.60:40519)
0808 04:09:44.348982 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:44.348988 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:44.360527 (+ 11539us) server_negotiation.cc:407] Connection header received
0808 04:09:44.360998 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:44.361005 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:44.361114 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:44.361188 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:44.362356 (+ 1168us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51250: BlockingRecv error: recv got EOF from 127.0.0.1:51250 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260808 04:09:44.365317 6370 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:09:44.357179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40568 (local address 127.5.83.62:40575)
0808 04:09:44.357396 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:44.357403 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:44.363245 (+ 5842us) server_negotiation.cc:407] Connection header received
0808 04:09:44.363812 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:44.363818 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:44.363903 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:44.363982 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:44.365050 (+ 1068us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40568: BlockingRecv error: recv got EOF from 127.0.0.1:40568 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260808 04:09:44.368413 6476 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:09:44.366211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53368 (local address 127.5.83.61:43419)
0808 04:09:44.366372 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:44.366377 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:44.366862 (+ 485us) server_negotiation.cc:407] Connection header received
0808 04:09:44.367115 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:44.367120 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:44.367195 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:44.367267 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:44.368161 (+ 894us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53368: BlockingRecv error: recv got EOF from 127.0.0.1:53368 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
I20260808 04:09:44.428864 6476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:44.404410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53384 (local address 127.5.83.61:43419)
0808 04:09:44.404607 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:44.404613 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:44.414754 (+ 10141us) server_negotiation.cc:407] Connection header received
0808 04:09:44.420858 (+ 6104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:44.420864 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:44.421094 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:44.421301 (+ 207us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:09:44.422445 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:44.423690 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:44.425760 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:44.426429 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:44.426527 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:44.427267 (+ 740us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:09:44.428028 (+ 761us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:09:44.428157 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:44.428419 (+ 262us) server_negotiation.cc:299] Negotiation successful
0808 04:09:44.428513 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260808 04:09:44.494493 6619 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:09:44.481626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51252 (local address 127.5.83.60:40519)
0808 04:09:44.481813 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:44.481819 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:44.492379 (+ 10560us) server_negotiation.cc:407] Connection header received
0808 04:09:44.492884 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:44.492891 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:44.493027 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:44.493104 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:44.494193 (+ 1089us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51252: BlockingRecv error: recv got EOF from 127.0.0.1:51252 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 04:09:44.522442 6476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:44.513579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53392 (local address 127.5.83.61:43419)
0808 04:09:44.513793 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:44.513799 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:44.516258 (+ 2459us) server_negotiation.cc:407] Connection header received
0808 04:09:44.516461 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:44.516467 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:44.516572 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:44.516690 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:09:44.517163 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:44.518336 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:44.520248 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:44.520984 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:44.521112 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:44.521604 (+ 492us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:09:44.521756 (+ 152us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:09:44.521816 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:44.522050 (+ 234us) server_negotiation.cc:299] Negotiation successful
0808 04:09:44.522142 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260808 04:09:44.533726 6619 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:09:44.529678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51256 (local address 127.5.83.60:40519)
0808 04:09:44.529888 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:44.529894 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:44.532260 (+ 2366us) server_negotiation.cc:407] Connection header received
0808 04:09:44.532418 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:44.532424 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:44.532527 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:44.532635 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:44.533510 (+ 875us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51256: BlockingRecv error: recv got EOF from 127.0.0.1:51256 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 04:09:44.546252 6476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:44.539201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53394 (local address 127.5.83.61:43419)
0808 04:09:44.539362 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:44.539367 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:44.540220 (+ 853us) server_negotiation.cc:407] Connection header received
0808 04:09:44.540495 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:44.540500 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:44.540603 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:44.540694 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:09:44.541134 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:44.542308 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:44.544257 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:44.544876 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:44.545007 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:44.545467 (+ 460us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:09:44.545597 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:09:44.545692 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:44.545859 (+ 167us) server_negotiation.cc:299] Negotiation successful
0808 04:09:44.545938 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260808 04:09:44.565992 6619 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:09:44.557384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51266 (local address 127.5.83.60:40519)
0808 04:09:44.557573 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:44.557579 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:44.564376 (+ 6797us) server_negotiation.cc:407] Connection header received
0808 04:09:44.564610 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:44.564616 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:44.564713 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:44.564815 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:44.565760 (+ 945us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51266: BlockingRecv error: recv got EOF from 127.0.0.1:51266 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260808 04:09:44.579762 6476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:44.573038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53406 (local address 127.5.83.61:43419)
0808 04:09:44.573263 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:44.573269 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:44.573741 (+ 472us) server_negotiation.cc:407] Connection header received
0808 04:09:44.573911 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:44.573917 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:44.574033 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:44.574148 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:09:44.574597 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:44.575769 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:44.577664 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:44.578380 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:44.578522 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:44.578970 (+ 448us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:09:44.579112 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:09:44.579218 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:44.579356 (+ 138us) server_negotiation.cc:299] Negotiation successful
0808 04:09:44.579438 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
W20260808 04:09:44.588601 6619 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:09:44.586453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51278 (local address 127.5.83.60:40519)
0808 04:09:44.586684 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:44.586690 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:44.587178 (+ 488us) server_negotiation.cc:407] Connection header received
0808 04:09:44.587324 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:44.587337 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:44.587427 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:44.587532 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:44.588385 (+ 853us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51278: BlockingRecv error: recv got EOF from 127.0.0.1:51278 (error 108)
Metrics: {"server-negotiator.queue_time_us":121}
I20260808 04:09:44.608419 6476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:44.601494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53416 (local address 127.5.83.61:43419)
0808 04:09:44.601667 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:44.601674 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:44.602247 (+ 573us) server_negotiation.cc:407] Connection header received
0808 04:09:44.602435 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:44.602445 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:44.602541 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:44.602652 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:09:44.603190 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:44.604426 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:44.606355 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:44.607004 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:44.607140 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:44.607605 (+ 465us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:09:44.607785 (+ 180us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:09:44.607889 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:44.608015 (+ 126us) server_negotiation.cc:299] Negotiation successful
0808 04:09:44.608102 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260808 04:09:44.616943 6619 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:09:44.614708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51286 (local address 127.5.83.60:40519)
0808 04:09:44.614947 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:44.614954 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:44.615480 (+ 526us) server_negotiation.cc:407] Connection header received
0808 04:09:44.615629 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:44.615634 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:44.615759 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:44.615868 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:44.616717 (+ 849us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51286: BlockingRecv error: recv got EOF from 127.0.0.1:51286 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
I20260808 04:09:44.648627 6476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:44.641479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53424 (local address 127.5.83.61:43419)
0808 04:09:44.641690 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:44.641697 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:44.642279 (+ 582us) server_negotiation.cc:407] Connection header received
0808 04:09:44.642580 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:44.642587 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:44.642677 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:44.642786 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:09:44.643297 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:44.644480 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:44.646376 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:44.647099 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:44.647230 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:44.647822 (+ 592us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:09:44.647969 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:09:44.648080 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:44.648229 (+ 149us) server_negotiation.cc:299] Negotiation successful
0808 04:09:44.648327 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260808 04:09:44.657166 6619 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:09:44.655115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51290 (local address 127.5.83.60:40519)
0808 04:09:44.655316 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:44.655322 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:44.655656 (+ 334us) server_negotiation.cc:407] Connection header received
0808 04:09:44.655903 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:44.655908 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:44.656017 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:44.656111 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:44.656951 (+ 840us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51290: BlockingRecv error: recv got EOF from 127.0.0.1:51290 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260808 04:09:44.698629 6476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:44.692732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53440 (local address 127.5.83.61:43419)
0808 04:09:44.692915 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:44.692921 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:44.693304 (+ 383us) server_negotiation.cc:407] Connection header received
0808 04:09:44.693473 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:44.693478 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:44.693562 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:44.693687 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:09:44.694037 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:44.695138 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:44.696751 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:44.697397 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:44.697542 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:44.697906 (+ 364us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:09:44.698027 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:09:44.698109 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:44.698228 (+ 119us) server_negotiation.cc:299] Negotiation successful
0808 04:09:44.698304 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260808 04:09:44.707931 6619 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:09:44.706175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51298 (local address 127.5.83.60:40519)
0808 04:09:44.706373 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:44.706378 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:44.706607 (+ 229us) server_negotiation.cc:407] Connection header received
0808 04:09:44.706765 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:44.706768 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:44.706837 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:44.706918 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:44.707682 (+ 764us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51298: BlockingRecv error: recv got EOF from 127.0.0.1:51298 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260808 04:09:44.779991 6476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:44.775414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53448 (local address 127.5.83.61:43419)
0808 04:09:44.775588 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:44.775592 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:44.775982 (+ 390us) server_negotiation.cc:407] Connection header received
0808 04:09:44.776133 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:44.776137 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:44.776205 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:44.776280 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:09:44.776624 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:44.777337 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:44.778651 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:44.779025 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:44.779120 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:44.779440 (+ 320us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:09:44.779534 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:09:44.779591 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:44.779683 (+ 92us) server_negotiation.cc:299] Negotiation successful
0808 04:09:44.779764 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260808 04:09:44.787230 6619 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:09:44.785472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51306 (local address 127.5.83.60:40519)
0808 04:09:44.785641 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:44.785647 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:44.785985 (+ 338us) server_negotiation.cc:407] Connection header received
0808 04:09:44.786146 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:44.786150 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:44.786219 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:44.786319 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:09:44.787033 (+ 714us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51306: BlockingRecv error: recv got EOF from 127.0.0.1:51306 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 04:09:44.879570 6591 catalog_manager.cc:1259] Loaded cluster ID: 0e4eea5ac4aa4d8e9762e0c1ec51022f
I20260808 04:09:44.879668 6591 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cb7eff48da4746c889ee7e9e791e3afa: loading cluster ID for follower catalog manager: success
I20260808 04:09:44.882169 6591 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cb7eff48da4746c889ee7e9e791e3afa: acquiring CA information for follower catalog manager: success
I20260808 04:09:44.883869 6591 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cb7eff48da4746c889ee7e9e791e3afa: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 04:09:44.924062 6476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:44.918939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53458 (local address 127.5.83.61:43419)
0808 04:09:44.919181 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:44.919188 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:44.919504 (+ 316us) server_negotiation.cc:407] Connection header received
0808 04:09:44.919728 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:44.919734 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:44.919813 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:44.919897 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:09:44.920326 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:44.921045 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:44.922460 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:44.922866 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:44.922978 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:44.923369 (+ 391us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:09:44.923482 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:09:44.923558 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:44.923724 (+ 166us) server_negotiation.cc:299] Negotiation successful
0808 04:09:44.923807 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260808 04:09:44.934489 6619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:44.929291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51318 (local address 127.5.83.60:40519)
0808 04:09:44.929453 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:44.929457 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:44.929825 (+ 368us) server_negotiation.cc:407] Connection header received
0808 04:09:44.929973 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:44.929977 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:44.930129 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:44.930220 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:09:44.930574 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:44.931276 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:44.932632 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:44.933047 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:44.933135 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:44.933569 (+ 434us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:09:44.934051 (+ 482us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:09:44.934156 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:44.934192 (+ 36us) server_negotiation.cc:299] Negotiation successful
0808 04:09:44.934250 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 04:09:44.947475 6619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:44.939921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51330 (local address 127.5.83.60:40519)
0808 04:09:44.940100 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:44.940106 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:44.941654 (+ 1548us) server_negotiation.cc:407] Connection header received
0808 04:09:44.941800 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:44.941805 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:44.941865 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:44.941937 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:09:44.942643 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:44.943531 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:44.945727 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:44.946186 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:44.946291 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:44.946733 (+ 442us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:09:44.946863 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:09:44.946948 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:44.947137 (+ 189us) server_negotiation.cc:299] Negotiation successful
0808 04:09:44.947210 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 04:09:44.948163 6476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:44.940780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53460 (local address 127.5.83.61:43419)
0808 04:09:44.940961 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:44.940966 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:44.942500 (+ 1534us) server_negotiation.cc:407] Connection header received
0808 04:09:44.942748 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:44.942753 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:44.942846 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:44.942937 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:09:44.943464 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:44.944255 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:44.945866 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:44.946476 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:44.946593 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:44.947147 (+ 554us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:09:44.947263 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:09:44.947347 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:44.947830 (+ 483us) server_negotiation.cc:299] Negotiation successful
0808 04:09:44.947908 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 04:09:44.948269 6817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:09:44.939856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40580 (local address 127.5.83.62:40575)
0808 04:09:44.941172 (+ 1316us) server_negotiation.cc:206] Beginning negotiation
0808 04:09:44.941178 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:09:44.941199 (+ 21us) server_negotiation.cc:407] Connection header received
0808 04:09:44.941317 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:09:44.941323 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:09:44.941539 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:09:44.941765 (+ 226us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:09:44.942727 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:44.943602 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:44.945742 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:09:44.946349 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:09:44.946479 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:09:44.946967 (+ 488us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:09:44.947407 (+ 440us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:09:44.947499 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0808 04:09:44.947930 (+ 431us) server_negotiation.cc:299] Negotiation successful
0808 04:09:44.948010 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1184,"thread_start_us":1059,"threads_started":1}
I20260808 04:09:44.953510 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 6276
I20260808 04:09:44.954387 6362 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:09:45.065856 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 6276
I20260808 04:09:45.091161 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 6377
I20260808 04:09:45.092309 6465 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:09:45.196074 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 6377
I20260808 04:09:45.220139 5452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2cd2tT/build/asan/bin/kudu pid 6479
I20260808 04:09:45.221244 6561 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:09:45.325315 5452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2cd2tT/build/asan/bin/kudu with pid 6479
2026-08-08T04:09:45Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4453 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9149 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (358663 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST